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ABSTRACT 


The Clinger-Cohen Act of 1996 requires performance measurement of 
information technology systems. Measuring the performance of program management 
for the Chemical Accoxmtability Management Information Network (CAMIN) system 
requires a thoughtful selection of useful metrics. The CAMIN is a complex Management 
Information System in the post deployment software system (PDSS) phase of the system 
life cycle. This research uses three primary sources for candidate metrics for a PDSS like 
CAMIN: 1) typical software metrics from DoD and commercial applications, 2) typical 
fielded software system metrics from DoD and commercial applications, and 3) case 
analysis of metrics currently used by CAMIN and other DoD systems in the PDSS phase. 
Analysis of these candidate metrics creates a concise list of combined metrics that are 
applicable to fielded software systems. The current primary issues of CAMIN program 
management establish the basis for selection of appropriate program management metrics 
from the candidate list. These issues are examined in a process to answer the primary 
research question, ” What are appropriate metrics and measures for management of the 
Chemical Accountability Management Information Network?” 
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I. INTRODUCTION 


A. PURPOSE 

This research identifies, evaluates, and selects the metrics for program 
management (PM) of the Chemical Accountability Management Information Network 
(CAMIN), an automated information system (AIS) that is in the Post Deployment 
Software Support (PDSS) phase. 

This new look at the program management metrics used for AIS in the PDSS 
phase is required to ensure that the system measures are appropriate to the phase of the 
system. Metrics provide the Program Manager (PM) tools to assess the program quality, 
trends, and requirements. During the PDSS phase, most systems undergo iterative 
development-type activities to maintain life cycle fimctionality. Systems also require 
concurrent maintenance and operations activities to optimize daily system operation. 

Published software metrics target assistance in managing activities specific to the 
software development phase. These metrics are applicable during the PDSS phase to 
support iterative maintenance upgrades, but the metrics used must also consider the 
issues of fielded systems. This research considers typical metrics used in management of 
software development, in combination with typical metrics that managers can use in the 
management of fielded standard systems. The research focuses specifically on metrics 
and measures for fielded software systems in consideration of their applicability toward 
the CAMIN. 
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B. BACKGROUND 


This study investigates the management and metrics of the CAMIN system. The 
CAMIN system is a database for accountability of chemical weapons (CW), former CW 
production facilities (CWPF), and other data that merits preservation in support of CW 
treaties. The system maintains its inventory controls like those used in a banking system, 
requiring transactions that fully document change of status, description, movement, or 
destruction of buildings, weapons, or equipment. The system purpose is twofold: 1) to 
enable compliance with Army wholesale and retail accountability regulations, and 2) to 
provide tracking and reporting required under the CW Treaty for short-notice inspections, 
continuous presence during destruction, and annual reporting obligations. The CAMIN 
supports users from specific organizations that have a CW-related mission wi thin the 
Department of Defense (DoD). 

The CAMIN system management utilizes the typical program management 
metrics of all programs. The program measures key areas like budgeting, funding, 
expenditures, performance, and schedule. This study addresses these common metrics, 
and considers other metrics that may be applicable to CAMIN. 

The field of AIS is constantly changing and expanding. The effective 
management of AIS requires the use of specialized management tools and techniques, 
relative to standard DoD systems. The primary differences in AIS relate to work force 
needs, costs, and turnover rates, scheduling and planning of creative breakthroughs, and 
the high frequency of external environment changes. 
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Most of the current literature on managing AIS focuses on the initial development 
and acquisition of systems. However, there is a dearth of information on supporting and 
maintaining fielded automation systems. Documentation within DoD and commercial 
organizations erroneously implies that the development effort “completes” the system. 

Fielded software systems require an accelerated iterative development. A 
manager can adapt to use developmental software metrics and measures. However, little 
information exists within DoD or commercial documents regarding the other 
management metrics that would be applicable for fielded software systems. 

The Year 2000 (Y2K) scare was indicative of the need for more vigilant 
management of fielded software systems. Managers found systems to be technologically 
backward, and months of effort were required to repair the flaws. A few systems 
required a prohibitively costly effort to repair, and managers had to replace the entire 
system. For example, developers are not available to correct programs written in 
programming languages that are no longer commonly used. When the language is not 
common, programmers necessary to update older systems are no longer available or cost- 
effective, and developers no longer support the system languages with updates. 

As managers field more AIS, the experiences that are now collected are valuable 
in managing future programs. This thesis documents the valuable information on metrics 
that managers can apply to the CAMIN system and other similar fielded software 
systems. 
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C. RESEARCH QUESTIONS 

1. Primary Research Question 

What are appropriate metrics and measures for management of the Chemical 
Accountability Management Information Network? 

2. Subsidiary Research Questions 

What is the CAMIN System, and how does its management occur? 

What are typical management metrics and measures for fielded systems, and how 
do managers perform management on fielded systems? 

What are Management Information Systems? What are the applications for these 
systems? 

What are typical management metrics and measures for Development of 
Management Information Systems, and how do managers perform management for 
Management Information Systems during development? 

What are the relevant acquisition phases and activities for Management 
Information Systems? In what ways are they similar to and different from those 
corresponding to other types of systems? 

What measures of effectiveness can help to assess Management Information 
Systems metrics and measures during PDSS? 

How can the results of this research be generalized? What lessons can be learned 
firom this analysis? 
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D. SCOPE OF THESIS, LIMITATIONS, AND ASSUMPTIONS 


This thesis addresses metrics for management of the CAMIN in the PDSS phase. 
It investigates lessons learned by managers of the CAMIN and of other similar systems. 
The thesis includes only systems in the PDSS phase that are managed within DoD. 

The metrics identified and evaluated in this thesis are within the criteria of those 
specifically associated with software programs and those associated with fielded systems. 
The common metrics literature for software systems are oriented toward development, 
and do not address measures that would specifically apply to software systems in use. 

This thesis defines the term “metrics” as a standard of measurement and the 
application of statistics and mathematical analysis to a specified field of study. The 
metrics described and defined in this thesis are specific ways of measuring and evaluating 
the defined aspects of the CAMIN program. 

E. METHODOLOGY 

This thesis methodology employs the case method. The case method includes 
three major elements of research. First, the thesis examines system management for the 
CAMIN system. Next, the thesis presents the results of a document search of common 
metrics that may be applicable to CAMIN. Finally, the thesis includes a study of metrics 
for similar systems. This research develops and analyzes data for applicability to the 
CAMIN Program. 

The thesis documents the analysis of these data and makes specific 
recommendations for the CAMIN program management metrics. The data are a 
compilation of common documented metrics for software development and program 
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management, together with management metrics for other fielded software systems. This 
analysis assesses metrics for applicability based on past, current, and anticipated CAMIN 
management issues. 

F. THESIS ORGANIZATION 

The first chapter of this thesis provides a general introduction of the purpose, 
background, objectives, the research questions that apply to this study, scope, 
methodology, and expected benefits of the research. 

Chapter II provides the background material to begin the research and analysis of 
this thesis. Following the introductory section is a detailed descriptive history of the 
CAMIN system, its missions, and functions, and how the organizational structure is set 
up to operate and manage the system. There is a discussion of the CAMIN system and its 
status with regard to the acquisition process. This chapter includes significant historic 
events and associated CAMIN program metrics. Finally, Chapter II defines and discusses 
the CAMIN program from the program management perspective concerning five basic 
issue areas. 

In Chapter III, the thesis discusses the t5^es of metrics that managers typically use 
in Management Information Systems. The chapter presents a detailed description of 
software system metrics identified through literature search. The types of metrics that 
managers typically use in program management of Fielded Systems are included. This 
chapter also addresses two fielded software systems that may be comparable to CAMIN. 
This chapter presents metrics identified through case study interviews of other fielded 
software systems. These data provide metrics and other da ta for analysis. 
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Chapter FV analyzes the data presented in the thesis. Chapter IV discusses 
observations from literature and other fielded software systems in reference to the 
CAMIN program areas of concern that may warrant metrics and measures. The 
observations create the basis for the final set of metrics for CAMIN. Data from the thesis 
data collection and analysis process generate lessons learned that are applicable to the 
subject. 

Chapter V provides conclusions and recommendations. In addition to the 
conclusions that one can draw from the results and analysis, specific recommendations to 
the CAMIN program and recommendations to other fielded software systems Managers 
are included. The final output of the study is a listing and description of future research 
topics. 

G. EXPECTED BENEFITS OF TfflS THESIS 

The results of this study provide direct benefit to the CAMIN Progr am The PM 
selected the metrics historically used on the program, but with thoughtful evaluation, the 
system metrics that this thesis selects can have a direct and meaningful effect on the 
program’s cost, schedule, and performance. Actively evaluating and selecting program 
metrics can help to guide future planning on the CAMIN system. The thesis should 
support future annual planning and management for the CAMIN System. 

Finally, managers can use these results in planning for the PDSS phase of future 
automation systems. This study results in useful information that automation system 
managers may find applicable in planning and establishing metrics and measures for 
other fielded software systems. 
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II. THE CAMIN SYSTEM, PAST AND PRESENT 


The CAMIN system program management is complex due to the system size and 
number of applications, and management of complex systems warrants measures. The 
program has a history of collecting and interpreting metrics. This chapter provides a 
descriptive history of the CAMIN system and the historic approach to program metrics. 
Understanding the history of the program is important to accomplishing the objective of 
this thesis. Every program is unique, and requires individual analysis to determine 
appropriate metrics for the program in a given phase of development. 

This chapter describes the CAMIN system, and its status with regard to the 
Acquisition Process. The CAMIN Program history has been very complex and unique, 
and this chapter includes a brief discussion of how CAMIN evolved through each of the 
acquisitions phases. 

This chapter concludes with a listing of the key program management issues for 
CAMIN, and a breakdown of factors that contribute to these issues. 

A. HISTORY OF CAMIN AND METRICS 

1. CAMIN Mission 

The CAMIN Program has been in existence for over six years. The CAMIN is a 
dual-purpose system, serving the chemical treaties for the DoD and the accountability 
mission for the Army’s CW stockpile under the DoD Single Manager for Conventional 
Ammunition. 
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Key areas derived from these two missions drive the CAMIN configuration and 
architecture. The Treaty mission must support short-notice Treaty inspections. Because 
of the sensitive nature of the Treaty data and the potential for impact on intemational 


relations, there is a zero-tolerance goal in the CAMIN for data errors relative to Treaty 


reporting and accountability of chemical materiel. There are CAMIN users at sites 
throughout DoD, and these users need to access real-time data 24 hours a day, seven days 
a week (24-7). System administration controls permissions to ensure that only users with 
training and responsibility can change the data. The CAMIN must provide the ability to 
audit all data changes, and adequate security can help to protect the data in CAMIN. 



CAMIN Data 



CA MIN 

DoD Treaty 
Accountability 

Destruction 
Reporting 


Tracking 
Tagged 
Items 

Declarations 
and Inspections 

Locally managed items 



[From Ref 15] 
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Figure 1. 




Figure 1 shows the types of data tracked in CAMIN, and how those data fit into 
the major mission areas of CAMIN. As shown here, data exist in CAMIN that do not fall 
into either area, such as data for locally owned stocks. Users choose to store and 
maintain these data in CAMIN for convenience. Additionally, this thesis uses the terms 
“destruction” and “demilitarization” interchangeably, in conjunction with elimination of 
CW assets. 

CAMIN provides tracking, accountability, and reporting required for DoD 
compliance with the Chemical Weapons Convention (CWC), [Ref 18] a Treaty formally 
known as the “Convention on the Prohibition of the Development, Production, 
Stockpiling, and Use of Chemical Weapons and on Their Destruction.” The CWC is a 
multilateral Treaty, governed from The Hague, by the Organization for the Prohibition of 
Chemical Weapons (OPCW). As of 12 February 2001,174 State Parties have signed the 
CWC, and 143 countries have ratified the treaty, including the United States [Ref. 19]. 
The CWC specifically addresses the destruction of CW and CWPF. 

The CAMIN performs all of the user requirements using a familiar, windows- 
based interface. Screen captures for typical representative CAMIN applications are 
available for viewing in Appendix E. 

CAMIN tracks locations of Treaty-declarable items within DoD at the 
geographical site, the declared area within the site, the building, and the grid within the 
building. The system tracks item information for the CWC such as nomenclature, serial 
number, and tags applied by inspectors. Documentation and information about historical 
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movements, destruction, or changes in item status reside in CAMIN. CAMIN maintains 
site diagrams for all declared facilities, as well as process flow diagramg of CWPF. 
CAMIN provides specially formatted reports for the short-notice Treaty inspections, 
while data from the system provides input to the annual reports submitted by the United 
States to the OPCW. 

CAMIN also tracks Schedule 1 chemical material that the DoD mflintaing for 
permitted purposes under the CWC [Ref 18]. The CWC monitors and controls three 
schedules of chemicals. The Schedule 1 list mainly comprises the weapons-grade toxic 
chemicals. Schedule 2 and 3 chemicals include precursors to toxic chemicals, which are 
toxic chemicals that are commonly used for industrial purposes. The CAMIN system 
does not currently track information related to Schedule 2 or 3 chemicals, as DoD does 
not have Schedule 2 or 3 chemical assets that are subject to CWC monitoring. The CWC 
permits State Parties to develop, produce, otherwise acquire, retain, transfer, and use 
toxic chemicals and their precursors for purposes not prohibited under the convention. 
These permitted purposes are industrial, agricultural, research, medical, pharmaceutical, 
or other peaceful purposes. In support of short-notice inspections, the CAMIN tracks 
items that the DoD retains for these purposes as individual items. CAMIN also maintains 
the chemical amounts for Schedule 1 permitted activities to ensure that the DoD remains 
in compliance with Treaty thresholds for maximum storage and production levels. 

In support of the CWC, CAMIN maintains auditable accountability records for 
CWPF equipment and for Schedule 1 permitted items. For CW, CAMIN also maintains 
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auditable accountability, not only for the CWC, but also in support of the Army’s 
accoimtability mission. 

CAMIN maintains strict accountability of all wholesale CW, chemical-peculiar 
equipment, and bulk storage of chemical materiel. CAMIN ensures compliance with 
U.S. Army and Army Materiel Command regulations that govern wholesale property 
accountability. CAMIN gives a full audit trail for change to status, nomenclature, codes, 
location, or destruction. CAMIN provides documents and application for performing 
annual inventories of the chemical materiel. Data in CAMIN include building lists, site 
diagrams, and movement and destruction records. CAMIN provides diagrams of the 
buildings and the specific grid locations that store items. The weapons and production 
equipment information in CAMIN includes not only location, but also description, serial 
niambers. Army codes for condition and ownership, and other parameters. 

In addition to its two primary designed purposes, CAMIN also permits system 
users to track locally owned stocks, defect codes, and other codes for munitions that are 
not required for CWC or Army wholesale accountability. 

2. CAMIN Management and Funding Structure 

The Defense Threat Reduction Agency developed the CAMIN system, and then 
transitioned program ownership to the U.S. Army Soldier and Biological Chemical 
Command in 1997. Presently, CAMIN program management resides within the 
Operations Enterprise of the Soldier and Biological Chemical Command (SBCCOM). 
The Stockpile Management Team performs functions required to manage the CAMIN 
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program, but the funding and approval process involves multiple organizations within the 
enterprise, as shown in Figure 2. 
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[Developed by Researcher] 

Figure 2. CAMIN Management Organization 

The two primary sources of funding for the CAMIN program are storage and 
treaty. There is a negotiated agreement for the distribution of funding requirements, 
between storage and treaty, for the Operation and Maintenance (O&M) Phase of the 
program. Before transition of Army wholesale accountability to CAMIN, the Treaty 
program funded the entire system. After transition, the “customer” who requires the 
system upgrade is responsible for funding the upgrade. In essence, the customer funds 
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the effort to accomplish its specific requirement. For activities of general benefit, such as 
user support and system administration, a basic allocation shown in Table 1 uses the 
number of active CW storage sites as a parameter to determine funding allocations. The 
Treaty portion of the funding for these general costs increases as CW Stockpile sites 
complete destruction activities. In this table, the term “CW Sites” refers to the number of 
active CW Stockpile Storage Sites during that fiscal year (FY). The number compares to 
the baseline total number of nine stockpile sites, to calculate the portion of funding for 
Treaty vs. storage. For example, Johnston Atoll Chemical Agent Disposal System 
(JACADS) completed destruction in FY 01, and in FY 02, there is one less CW site to 
support. These numbers, based on current destruction schedules, are subject to revision. 
Understanding this funding and management structure helps to understand the metrics 
that this program must collect. Only by defining appropriate metrics can the program 
justify adequate funding to maintain system operations. 
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[From Ref. 14] 

Table 1. CAMIN Funding Allocation 
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personally involved with the issuance of storage funding. Within the Operations 
enterprise, three organizations are involved in CAMIN management and funding: 1) The 
Stockpile Management Team, 2) the Center for Treaty Implementation and Compliance 
(CTIC), and 3) the Business Management Team (BMT). 

The Stockpile Management Team (SMT) provides the overall CAMIN 
management and serves as the National Inventory Control Point and Accountable Officer 
for CW and Chemical-Unique Containers. The team leader participates in the budgeting 
process for storage funding. 

A team acting as a program management office (PMO) performs CAMIN 
management functions Avithin the SMT. The PMO for CAMIN performs business 
planning and budgeting for the CAMIN program. The majority of funding for CAMIN 
goes into the support contract, for user support and training, system administration, data 
administration, hardware and software purchase, software maintenance and support. In 
addition to business planning, budgeting, planning, and execution, CAMIN program 
management functions performed by Government personnel include help desk 
configuration management, data management, and data administration. The program 
areas that follow provide detailed descriptions of these tasks. 

As stated above, the National Inventory Control Point (NICP) and Accoimtable 
Officer missions for CW and Chemical-Unique Containers missions reside within the 
Stockpile Management Team. In addition to its role as an active user of CAMIN, the 
NICP/Accountable Officer also provides policy guidance for the CAMIN program in the 
areas of storage and accountability. Additionally, when the CAMIN program needs 
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Dollars in Thousands ($K) 


funding to satisfy a new or modified requirement, this organization provides storage 
funding approval to the decision makers for storage funding within the Operations 
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Figure 3. CAMIN Funding History 



































The CTIC performs Treaty Management for SBCCOM and for the CAMIN 
program. They provide Treaty Policy Support and Treaty Funding. They also act as 
Configuration Control Board (CCB) Co-Chair. When the CAMIN program needs 
funding to satisfy a new or modified requirement, this organi 2 ation provides approval for 
Treaty funding, as well as the funding. 

The BMT witbin die Operations Enterprise manages all funding that comes into 
the enterprise. The BMT distributes all funding that the CTIC later allocates to the 
CAMIN Program. The BMT manages the storage funding, and only issues this funding 

to the CAMIN program with authorization from die Operations Enteiprise Director and 
theNICP. 

The chart of CAMIN Funding Histoiy [Figure 3] shows the funding levels and 
sources. The CAMIN program has never successfully conformed to the funding 
allocation profile that comprised the agreement between storage and Treaty in FY 99. As 
a result, managing funding for the CAMIN program requires intense coordination every 
year. The PMO must determine how much funding is required, request funding based on 
allocation agreement, and renegotiate based on actual funding allocated from the frinding 
sources. 


3. CAMIN System Description 

The CAMIN program is a stand-alone system, designed to main tain mission data 
and allow data retrieval by all authorized users on a 24-7 real-time basis. CAMIN 
management strictly controls user access permissions to ensure that data remain as 
accurate as possible. 
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CAMIN is a client-server system, where the CAMIN server is centrally located 
with CAMIN server software, and users operate the system on client software installed on 
remote workstations. The CAMIN Server is located at SBCCOM, Aberdeen, MD. 
Workstations are located at Army and contractor sites in the continental United States 
and the Pacific. The client platforms for CAMIN use Microsoft Windows™ to provide a 
familiar interface for the user community. The CAMIN server operates as a Web server 
to allow users, which have proper permission, to access a limited set of CAMIN 
functionality through a web browser interface. 

The CAMIN Architecture design depicted in Figure 4 provides access to data 
while maintaining security of the system and data. The primary areas that can contribute 
to the system vulnerability are the server, connectivity, and workstation. CAMIN has a 
published Continuity of Operations Plan that addresses these issues. 

The CAMIN Server is a highly available cluster design, with two mirrored 
servers, and the identical data set resides in both units. If the active server unit goes 
down, the system automatically switches over to the other server unit in the cluster and 
continues operation. Power outages at the server location are a source of concern, and 
the server hardware suite includes uninterruptible power supplies to smoothly shut down 
the server in case of power outage. The building in which the server resides is vulnerable 
to power outages and air conditioner failures. One preventative measure against future 
outages is air conditioner system replacement. The system administrators back up 
CAMIN data to a remote location daily and the system administrator can access backup 
data from the remote location if there is a systemic power failure at SBCCOM. 
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Precautions can be justified though experience. The server cluster experienced a 
systemic and long-term power failure at the server location, caused by hurricane Floyd in 
September 1999, while one of the client sites was trying to prepare for a short-notice 
Treaty inspection. The system administrator used data backups to meet the immediate 
need, but plans are now in place to help mitigate future problems. 



[After Ref. 14] 

Figure 4. CAMDST System Architecture 


Connectivity is another area of vulnerability. Consequently, there are multiple 
ways to provide access between CAMIN server and workstation, to include both Local 
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Area Network (LAN) and phone for the workstation client software, and web access. 
The other issue of connectivity involves the implementation of firewalls for security. 
Firewalls at server and client locations can block user access to the CAMIN server. 
Firewalls can block partial or total connection, depending on firewall settings. The 
CAMIN manager periodically monitors firewall settings and advises local Corporate 
Information Offices (CIOs) at firewall sites to ensure that CAMIN data and other 
required information can pass through the firewall. The CAMIN server is located on the 
network within an area of the SBCCOM firewall known as the DMZ, a “demilitarized 
zone” that is not completely within the firewall, but within a protected area at the firewall 
interface. This location permits users to access the system while re taining well-defined 
security benefits. 

The workstation and client software can also contribute to CAMIN vulnerability. 
The CAMIN workstations must have a special configuration to operate CAMIN client 
software, including a specific and atypical operating system, Windows NT 4.0. The 
installation of client software must be correct to allow CAMIN functionality. Other 
pieces of software or hardware installed on the workstation can interfere with CAMIN 
functionality. Users must gain approval from the CAMIN management team before 
installing new software or hardware onto their CAMIN workstation. For example, one 
CAMIN user loaded an unauthorized screen saver onto her workstation, and the screen 
saver increased the default font on her workstation so that CAMIN data did not appear in 
the text boxes on her workstation. This change was very difficult to diagnose and detect. 
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CAMIN authorized users can also access CAMIN data through the recently 
introduced web site. Users now have web access to frequently used CAMIN data reports. 
An expansion of web capability is planned that allows users to perform all workstation 
functionality through a web browser-based interface. This action resolves all workstation 
configuration, cost, and maintenance issues by eliminating the need for stringent 
workstation requirements. The web design distills the firewall connectivity issues to 
configuration of one common web port. 

4. CAMIN Program Areas 

CAMIN program management includes short and long-term program budgeting, 
business and strategic planning, and organizational and contractual activities. Program 
management also includes oversight of the areas of user support, O&M, and requirements 
upgrades for the CAMIN system. The CAMIN Management Team within the Stockpile 
Management Team is responsible for all areas listed in Table 2. 

a. Overarching Management 

CAMIN overarching management includes budgeting and execution, planning, 
reporting, and concept design, data management, contract management, configuration 
management, and hardware and software acquisition. The budgeting and execution 
process explanation is contained in the previous section. 
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[Developed by Researcher] 


Table 2. CAMIN Program Management Areas 

Planning, reporting, and concept design encompasses a large scope of 
activities. Generating strategic and business plans requires a study and application of 
trends in information technology, and development and application of metrics. The PMO 
passes on important information to Treaty and accountability management and through 
the organizational hierarchy to the DoD level. Policies for the CAMIN program establish 
consistency across users, workstations, and within SBCCOM, Army Materiel Command 
(AMC), and Army. 
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Data management includes all work toward improving the accuracy of 
data in CAMIN. The NICP, system users, and CTIC are involved in the process to 
identify and analyze data errors. Users or data administration accomplish data 
corrections. Defining the cause of the errors leads to corrective actions, including new 
problem/change requests (PCRs), enhanced training, and enhanced user manuals PCRs 
document requests for changes to the CAMIN software or system. Users of CAMIN 
software or data generate the PCRs that the CCB evaluates for disposition. Data 
management and administration activities are important, expensive, and time consuming 
to CAMIN Program Management. Challenges to the CAMIN data come through user 
error, data migration from one version to the next, and through policy changes. CAMIN 
maintains a strict audit trail of data changes, including documentation of the user that 
caused the change. Additionally, CAMIN uses quality assurance (QA) by requiring a 
second user login to “QA” the data change before making the actual data change in the 
system. Annual inventories and Treaty inspections further validate the data. User error 
requires frequent audits. However, proactive measures are used to find these errors 
before the external community becomes aware of them. However, data migration errors 
are difficult to identify before they occur. Policy changes often affect data, and the 
associated problems are targeted through analysis and testing. 

The contract management area ties closely to budgeting, funding 
execution, planning, and upgrades. The CAMIN contract is comprised of seven phases. 
Phase 1 is User Support, Phase 2 is System Maintenance, Phase 3 is Server Maintenance, 
Phase 4 is Meetings, Phase 5 is Special Studies, Phase 6 is Hardware and Software 
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Acquisition, Configuration, and Fielding, and Phase 7 is System Version Upgrades. 
Funding sources provide incremental fimding, and this makes planning difficult. The PM 
gives the contractor guidelines to establish which tasks to devote their time, and how to 
establish a workforce. Retention of Information Technology (IT) workers in the current 
hiring environment requires consistent funding and interesting and consistent work. 

Configuration management controls software, hardware, data, and 
documentation for the system. The configuration management plan defines roles and 
responsibilities. The system configuration manager is the PM. There are two co-chairs 
of the CCB, representing Treaty and accoxmtability/storage as the two missions of 
CAMIN. The CCB reviews PCRs from data and system users, and has the authority to 
approve system changes. The CCB consists of representatives fi-om funding, users, and 
other primary CAMIN customers. The CCB makes decisions based on sound business 
criteria. Criteria for analysis include the need and benefit of a change, the risks, the 
potential funding availability, the costs, and the difficulty/time required to introduce the 
change. 

Hardware and Software Acquisition requires forward planning. The 
workstation acquisitions tie to the three-year maintenance agreements and to the 
fluctuating workload associated with the destruction effort. Active destruction at each 
major facility increases the workload (and workstation) requirements by about five. The 
acquisition of workstation hardware and software ties to the support contract. The 
contractor configures the workstations, loading software and ensuring functionality 
through phone and LAN connection before delivery. Figure 5 shows the location of 
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connection before delivery. Figure 5 shows the location of CAMIN users. Appendix B 
provides a detailed listing of current CAMIN users and locations. 

b. User support 

User support covers help line, training, user group meetings, web notices, 
and announcements. User competence is an area that is critical to the CAMIN system, to 
help ensure that CAMIN data is correct and timely. The user group is geographically 
diverse, users are located all across the country [Figure 5]. The education and experience 
level of users vary from SES and frill Army Colonels to GS-05 clerks that have only used 
a computer for CAMIN. Help line support is available during working hours, and a 24- 
hour emergency pager is available to support after-hour emergencies. Scheduled training 
classes are offered both on-line and in a classroom-style training environment. 

Each version of the software includes all relevant docmnentation, 
including a full user manual loaded onto the CAMIN desktop. An on-line training 
database within the CAMIN system allows users to practice an activity and evaluate 
results against expectations. Users can switch to the training database to test or practice a 
process before affecting the live database. An additional training tool offered is the 
computer-based training CDs. Annual Data Management User Group meetings invite all 
users to participate in meetings to discuss the current and future activities of the CAMIN 
Program. Breakout sessions with user subgroups provide an additional venue for contact 
and feedback with iisers. Finally, local administrators at workstation locations manage 
the workstation configuration, including permissions and access for users, and settings 
for the local networks and firewalls to permit CAMIN access. 
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[After Ref. 14] 

Figure 5. CAMIN Workstation Sites and Users 


c. Administration 

Operation of the CAMIN system includes system administration and 
security activities, data administration, network and firewall, web administration, and 
workstation administration. System administration manages and maintains server 
operations on a daily basis, managing file sizes and system operations. The system 
administrator issues and maintains user accounts, passwords, and permissions. Security 
is of primary concern. Althou^ data in CAMIN is not classified, much of it is For 
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Official Use Only. Passwords and permissions control system access. CAMIN uses a 
commercial off-the-shelf (COTS) middleware software package called Distributed 
Computing Environment (DCE) on the workstations to provide access controls for client 
security. The client connections pass over the NIPRNET, an Army version of the 
Internet. For the web, Personal Key Identifier (PKI) and Secure Socket Layer (SSL) 
maintain a secure connection and encrypt data passing over the web. 

Data administration in CAMIN is required for correc ting data errors, 
making top-level changes to data, and for collecting metrics about data in the CAMIN 
system. The CAMIN architecture is set up so that system users have no direct access to 
the CAMIN database. The CAMIN system is extremely complex, with over 800,000 
lines of software code, and 3.8 Gigabytes of data files. Table 3 summarizes CAMIN 
lines of code, and Appendix C contains information that is more detailed. Direct access 
to the CAMIN database to change system data is limited to three people. The data 
administrators use a strict control process to monitor and approve da ta changes. 

As stated in the previous section, network and firewall issues can 
contribute to the vulnerability of the CAMIN system. Coordination with local 
administrators helps to ensure that network and firewall configurations are compatible 
with CAMIN. The CAMIN web site resides on the CAMIN server cluster. Web site 
administrators maintain links, post announcements, maintain the calendar, and ensure 
connectivity. 
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[After Ref. 4] 

Table 3. CAMIN Version 8.0 Lines of Code 

Workstation administration is labor-intensive. Due to the frequency of 
problems, sites assign a local workstation administrator at each installation. These 
administrators are not always well trained or qualified to do the work, but the individual 
provides CAMIN Managers with a single point of contact to send documentation and 
updates, as well as a single point of responsibility for monitoring the configuration of 
workstations. This effort is necessary because users and administrators load unapproved 
software and hardware, forget passwords, delete or overwrite files, dynamically linked 
libraries and registry, and otherwise create an environment that prevents CAMIN 
fimctionality on the workstation. DCE facilitates system fimctionality by preventing 
connectivity unless there is synchronization between the workstation and the server. 
Lack of workstation use to access CAMIN can cause the clock to lose synchronicity with 
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the server, and require the user or local administrator to clobber and reconfigure DCE 
before using CAMIN applications. 

d. Upgrades 

The Maintenance and Requirements Upgrades include the areas of Design 
and Development, Code Writing, Testing, and Fielding. These upgrades occur in the 
form of version changes or patches. Version changes have historically occurred one or 
two times per year, and patches may occur more fi-equently. The Problem/Change 
Request (PCR) Database keeps track of all PCRs, classified by CCB disposition, 
magnitude of difficulty, ability to accomplish in a patch, and whether the PCR is 
associated with a version or a patch. Version Upgrade Process: The process starts with a 
change being approved by the CCB, and with related requirements for design or software 
changes. The PM determines which changes are needed, and initiates funding requests 
and a contract modification to implement the upgrade. Award of the contract 
modification includes a negotiation of the exact number and combination of PCRs, and 
the schedule to accomplish these system modifications. The contractor proceeds with 
design and development, code writing, and then performs testing. The contractor 
provides a test area for the Government to perform acceptance testing. After Government 
acceptance, the contractor publishes and mails CDs, installation manuals, and Version 
Description Documents (VDD) to workstation administrators for dissemination and 
oversight of the installation. 
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[Developed by Researcher] 

Figure 6. CAMIN Maintenance Upgrade Activities 

Patch Process: An urgently needed change to the system typically drives 
the patch. The PM decides candidate PCRs, and then authorizes the contractor, through 
the Procuring Contracting Officer, to implement the patch. The contractor and PM test 
the patch, then make the patch available to download and install from the web site. The 
local workstation administrators are responsible to ensure that the workstations under 
their purview have the new patch installed. 

5. CAMIN Metrics 

The CAMIN PM uses metrics to assess the program in terms of schedtile, budget, 
and functionality requirements. The primary sources for these metrics are the contractor 
output, PM data from internal documentation, and the CAMIN system itself. The metrics 
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used are those easily accessible, those that are most obvious, and those that are required 
by upper management. 

The support contractor tracks additional metrics that address requirements, 
program management, quality assurance, configuration management, training, peer 
review, critical resource, maintenance, development, and intergroup coordination. 
Appendix D provides a more detailed list, summarizing the metrics tracked by the 
contractor. Documentation and process quality are very important to the contractor’s 
internal procedures. Their activities for the CAMIN program are qualified for between 
level 3 and 4 on the Software Engineering Institute (SEE) Software Capability Maturity 
Model (SW-CMM) evaluation. 

The PM uses metrics that address areas that are common with the contractor. In 
addition, the PM uses metrics to address contract management. The PM generates a 
weekly CAMIN report to measure and report progress of the CAMQN data against Treaty 
and destruction goals. The data in the weekly report provide measures of data and the 
number of transactions against those data for the weekly period. Another way that 
performance is measured, is to annually review the programmatic goals in the business 
plan against accomplishments for the year. 

The CAMIN program generates and utilizes metrics summarized in Table 4. The 
selection and application of appropriate metrics requires organization and thoughtful 
consideration. Each metric consumes time for collection, reporting, and analysis. 
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1 Metric 

Purpose 

Fxmding 
Obligated and 
Disbursed 

Tracking funding helps to schedule work, to help predict future 
shortcomings, to manage work force issues, and to help assess money 
expenditures. 

Schedule 

Tracking schedule helps to meet user needs and to control costs. 

User Help 

Tracking the type, duration, and somce of user help calls has benefited 
the program through identifying systemic problems. Evaluating help¬ 
line calls can drive a new training need or drive a change to the 

CAMIN design. Particular users or user types may need additional 
training. The user manual may need improvement. 

Data 

Interventions 

Data interventions are often expensive, time-consuming, and risky to 
the database. Those that are most skilled in database administration 
perform data interventions. Evaluation identifies systemic problems 
that may drive changes to user training, user manual, and/or changes to 
the system software. 

Percent 

Completion 

When developing a new version of software, track the level of 
completion of each phase. 

Training 

Training factors tracked include student list, types of training given, 
and feedback reports from trainees. These measure the adequacy of 
training through feedback and repeat trainees. 

Requirements 

A database tracks all existing requirements for CAMIN. If a 
requirement was completed, the database retains the date/version of 
completion. This metric provides a series of historical baselines, and 
tracks remaining work. 

PCR 

A database tracks PCRs for the CCB. This maintains the list of the 

PCRs, documentation of CCB discussion or decision on the action, and 
other related data elements. 
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Actions 

A database tracks the actions from meetings, the individual, or 
organization responsible, and completion information. This provides a 
measure of activity that affects cost, schedule, and performance of 
CAMIN. 

User List 

A list of users, their passwords and permissions, are tracked, and 
periodically assessed to keep the user list limited to those who need 
permissions, and to limit the permissions to only those needed. This 
control protects the CAMIN system and its data. 

Hardware/ 
Software List 

A list of program hardware and installed software helps to ensure that 
workstations are maintainable, and that they have the correct version of 
CAMIN and COTS software. 

User Accounts 
and Logins 

The CAMIN software has an application that provides reports of user 
accounts and logins. Through this, system usage and application usage 
is tracked. The data helps to determine if a users accounts or 
permissions need modification. The data can help determine if 
applications are under or over utilized. 


[Developed by Researcher] 

Table 4. CAMIN Program Management Metrics 


B. CAMIN AND THE ACQUISITION PROCESS 

CAMIN is in the Operations and Support Phase of the program. For automation 
systems, this phase corresponds to fielded software systems. Like typical software 
systems, the CAMIN goes through an iterative process of development, production, and 
deployment of system maintenance upgrades and enhancements. This process of cyclical 
changes to the system is often confusing to anyone that is not familiar with software 
systems. 

The DoD Acquisition Model (Figure 7) shows the basic phases of the acquisition 
process. This section provides a brief description of the history of the CAMIN Program 
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and significant events within the acquisition process framework. It is important to study 
the history of CAMIN versions in order to understand that CAMIN development has 
continued well into the PDSS phase. Each version update has added to the fimctionality 
of CAMIN to meet the original functional requirements, while making incremental 
enhancements to the functionality based on user needs. Version upgrades continue to be 
required through the life of the system. 

1. Concepts and Technology Development 

The developers spent little time on the Concept and Technology Development for 
CAMIN. In 1994-95, the DoD created an urgent requirement to develop a system for 
tracking, verifying, and reporting Chemical Treaty required data. The basis of this urgent 
requirement was the imcertainty of when the United States Senate would ratify the first of 
the pending chemical treaties. While common, urgency is never a good environment to 
develop software, and this, ultimately, caused long-term problems. 

The CAMIN developer, the Defense Threat Reduction Agency, then Defense 
Nuclear Agency, had an existing contract with TRW, then BDM Federal, to develop the 
Compliance Management and Tracking System (CMTS) for all DoD Treaty data. 

The CAMIN design uses the experience of two prior existing systems. The 
CAMIN system was initially an add-on to CMTS. Therefore, developers used CMTS as 
the template for system architecture and initial design of CAMIN. The other system 
involved in the development process was the existing Accoimtable system, the Toxic 
Chemical Munitions (TCM) Database. The TCM was insufficient to meet Treaty needs, 
and was not robust enough to continue operation as destruction began and transactions 
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increased significantly. The TCM design was very limiting. The language was dBase 
III, and it used a series of stand-alone workstations. TCM users transferred local 
accoimtability data to the Accountable officer periodically over secure phone lines. 
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When the developers began working to merge the architecture of CMTS with the 
fimctionality of TCM, the results became very complex. Due to complexity and unique 
accountability requirements, the system concept eventually grew into a system that is 
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completely separate from CMTS. CAMIN was now ready to enter the System 
Development and Demonstration phase. 



[From Ref. 14] 

Figure 8. CAMIN Program History 

2. System Development and Demonstration 

The System Development and Demonstration of the CAMIN included basic 
software development, with user involvement. The developers held a series of User 
Group meetings to discuss concepts and ideas. The developers created a proposed 
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architecture and presented it to users. They also presented sample screens and views to 
show the users how the system would look. 

The urgency for Treaty needs led to an interim goal for CAMIN to provide only 
minimal functionality. In this first version, applications had no inherent process other 
than to convert inputted data into the required format. As time passed and managers 
grew more frantic, the Treaty hierarchy decided to release a first version that addressed 
only the CW and accountability. This decision resulted in the delay in full meeting 
requirements for CWPF and Permitted Activities applications until later versions. 

The development of CAMIN was truncated, and Version I.O fielding occurred at 
three beta sites. Version 2.0 contained additional requirements and incorporated user 
feedback fi-om Version 1.0. The transition from R«&:D funding to primarily O&M 
funding occurred in this timefi-ame. 

3. Production and Deployment 

Developers fielded Version 2.0 to all user sites. In versions 3.0 and 4.0, the 
developer included additional functionality. With the new functionality, the system came 
closer to meeting the original system requirements. With version 5.0, the transition 
process began with the movement of contract management activities from the Defense 
Threat Reduction Agency (DTRA), as the Research and Development agency, to 
SBCCOM for Operations and Maintenance. The transition process included a full 
Independent Verification and Validation Test (IV&V), using an external agency to design 
and perform the test. The schedule in Figure 8 shows the transition period, spanning 
fi-om FY 98 through FY 99. During transition, SBCCOM awarded a sole source contract 
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to TRW for O&M of CAMIN. The developer transitioned the system in December 1998, 
Avith a couple of outstanding actions. Development of Computer-Based Tr aining (CBT) 
CDs by DTRA enabled multimedia system training. In addition, DTRA would also fund 
the required testing to accomplish Y2K requirements imposed by DoD. Once CAMIN 
replaced TCM to become the Army accountable system in 1998, storage funding became 
part of the overall funding for the management of CAMIN. 

4. Operations and Support 

The Operations and Support phase of CAMIN encompasses a wide variety of 
program management activities, as described in this document. In addition, new software 
versions continue to meet the changing needs of the users, the hardware/software/network 
environment, and the DoD. System and data users continue to identify new and refined 
requirements for CAMIN functionality. Changes included new data output or input 
formats. These types of changes to requirements can affect the structure of the database 
and cause significant revisions to the system. The environment changes also drive 
CAMIN changes. Firewalls and other network issues can require firequent system 
revisions. It is not practical to adopt new software versions as soon as the industry makes 
them available, but the CAMIN system must eventually migrate to receive the support 
,and to remain compatible with the external environment. The DoD has recently imposed 
significant security-related policies that affect the CAMIN workstations, server, and web 
site, as well. 

The CAMIN Version 8.0 fielding occurred in FYOl. A significant list of 
approved PCRs is ready to incorporate into the next version of CAMIN. Once funded. 
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the program to reduce the dependence on CAMIN workstations by developing and 
providing web-enabled applications for data entry and data access users can begin. 

C. AREAS OF INTEREST FOR CAMIN PROGRAM MANAGEMENT 

To summarize this chapter, the major issue areas of CAMIN program 
management provide a basis for analysis of metrics. The issues listed in this section are 
interrelated, but each is a distinct area for program management concern. It is important 
to view program management metrics in terms of program issues. The selected metrics 
must address all of these key areas. 

1. Funding 

As previously discussed, funding levels for the CAMIN Program have been 
unpredictable, but have reduced over the last two years. The funding deficit and 
uncertainty cause significant problems for the CAMIN system. Improved 
commumcation and controls mitigates the funding problems. However, measures of 
funding are needed to address the following areas: understanding the funding needs based 
on program management requirements, communication of fimding justification to fund 
managers, and funding availability to meet program requirements. 

2. System Availability 

System availability is critical for two key reasons. First, users must be able to 
access and update CAMIN data in support of short-notice Treaty inspections. 
Availability is critical to enable users to accomplish timely input of inventory 
transactions. Availability is dependent on connections, configuration, and power. To 
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meet PM requirements for system availability, areas that need to be addressed are: system 
software capability to perform customer requirements, power to the system, coimectivity 
through the network, firewall, and other security systems, workstation configuration and 
user account maintenance, and system maintenance to keep the system running 
efficiently. 


3. Data and Output Accuracy 

The critical missions of Treaty and accountability drive the need for data 
accuracy. Treaty requires accuracy to help maintain U.S. integrity with the international 
inspectors. The accountability mission requires a system with substantial data protections 
and an audit trail, with the aim of achieving 100% accuracy. Accuracy relates to the 
system design, to the user permissions, and to the competency of system users. An 
oversight organization also needs to periodically audit the data. In this case, both Treaty 
and accountability should audit data for their respective requirements. Factors in this 
process are listed here; training of users, help line for users of the system and data, user 
manuals, ease of use and data protections in CAMIN software, data audit by cognizant 
organizations, and data interventions to correct user errors and to perform mass changes 
to system data. 


4. Experienced IT Support 

Retention of IT support staff is important to reduce the extended learning curves 
associated with working on complex systems. The program’s ability to retain developers 
and maintainers of CAMIN is dependent on consistent funding, consistent work, and 
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overcoming technological challenges. The CAMIN program has sufficient work 
requirements if funding were available. Consistent workload through iterative 
maintenance activities can help to retain an experienced workforce. However, 
maintaining the consistent workforce involves leveling of tasks to achieve a consistent 
level of work, challenging the staff by using current or leading technology, and using a 
contract vehicle that provides adequate funding and work levels. 

5. Requirements Changes 

The area of requirement definition and associated changes is critical to continued 
CAMIN functionality and user satisfaction. Through the history of CAMIN, these factors 
have driven the program changes. The hardware and software require periodic upgrade 
to maintain continued operation and maintenance. The users and customers have added 
requirements and enhancements each year to reduce processing time, reduce errors, and 
improve the system. In order to assess revisions to system requirements, the following 
areas must be addressed: changes to COTS and hardware, network changes, state of the 
art technology and architecture, and evaluating changes in terms of time to accomplish 
change, and the priority and complexity of these changes. 
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III. DATA PRESENTATION: CANDIDATE METRICS 


Metrics have become commonplace in program management, and metrics tools 
are available through literature and through case analysis. In this chapter, data collection 
and analysis provides a thorough list of the types of metrics that may be appropriate for 
CAMIN and other systems in the PDSS phase. This chapter also contains a discussion of 
the types of metrics that managers may use in Management Information Systems in 
general. The term “Management Information Systems” refers to systems designed to 
automate business processes, rather than software systems embedded in weapon systems, 
vehicles, and the like. 

The metrics for management of software that exist in the current literature are 
oriented toward the development of software, rather than sustainment of deployed 
systems. When the CAMIN system undergoes its iterative software changes, the 
development-related metrics remain valid. However, CAMIN is a deployed system, and 
may warrant measurement in terms of customer satisfaction, user accessibility, including 
readiness and downtime, help desk support, and other feedback tools. Therefore, Chapter 
in also discusses of the types of metrics that Program Managers typically use in the 
management of fielded systems. 

The discussion of metrics in this chapter considers both DoD system metrics and 
commercial metrics. Use of commercial best practices is a tenet of the DoD 5000 
guidance on acquisition reform within DoD, and so consideration of Government and 
commercial practices should help to arrive at the optimal metrics selection. The chapter 
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also examines the metrics of other fielded software systems that are comparable to 
CAMIN, based on the life cycle, fimding, management style, and type of system. 

A. SOFTWARE METRICS 

The Clinger-Cohen Act, otherwise known as the Information Technology 
Management Reform Act of 1996 [Ref 5], provides tools for management of IT within 
the U.S. Government. These tools help to manage funds and contracts to acquire IT more 
efficiently by providing flexibility in the acquisition process. The Clinger-Cohen Act 
requires the use of measures for software acquisition and management specifically, as 
follows: 

The head of an executive agency shall (3) ensure that performance 
measures are prescribed for information technology used by or to be 
acquired for, the executive agency and that the performance measurements 
measure how well the information technology supports programs of the 
executive agency. 

[From Ref 5, Section 5123(3)] 

To establish valid metrics for software development and production, managers 
need tools to establish overall process quality. PMs can require that software developers 
certify the quality of the software production process through standards established in 
ISO 9000 and the SEI Software Capability Maturity Model (SW-CMM) [Ref 20]. Yet, 
software performance parameters are difficult to define and measure, as the definition of 
parameters for a system changes based on platform, progr amming language, and user 
interface. Additional factors in software measurement include the fast-changing 
environment in which the software must remain functional. The changes in the platform 
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and networks are fluid. Requirement creep is also more common with software-intensive 
systems than with strictly-hardware systems. In addition, integration testing often 
requires completed software and hardware. The metrics associated with successful 
systems development projects in successful companies are those that deliver on schedule, 
relatively close to budget, and with high levels of quality and reliability. 

1. Management Information Systems 

This thesis makes a differentiation between Management Information Systems 
and other systems. Systems compared here have more similarities than differences, 
especially when analyzed within the DoD acquisition framework. However, the 
differences have an effect on decisions about how to plan for and to manage these 
systems. Management Information Systems require frequent changes due to dependence 
on platform and COTS. Table 5 shows a comparison of the key program management 
parameters for each of the three program types. These differences are generalizations, 
but are typically true. 

Table 5 shows that requirements are very unstable for software systems, relative 
to hardware. External sources, such as environment changes, user needs, COTS changes, 
and platform changes drive modification to the system requirements. Development is 
iterative for software-based systems. Complex systems may experience iterative 
development for many years. Additionally, development-like activities occur as part of 
the process to perform maintenance upgrades to the system to meet changing system 
requirements. 


45 



Management Embedded Software Standard Hardware 
Information Systems Systems Systems 


Requirements 

Changes through the 
system life, driven by 
platform, 

environment, users 

Changes through the 
system life, driven by 
users and interface 
(more stable platform 
and environment) 

Theoretically frozen 
at ORD, really frozen 
at materiel release 

Development 

Iterative, costly, 
through the life of the 
program, dependent 
on experts and 
creativity 

Somewhat iterative, 
but limited by 
hardware platform 

One time- Waterfall 
to Production 

Testing 

Beta and IV&V 

DT and OT 

DT and OT 

Production 

Print CDs 

Install in production 
hardware 

Production may occur 
over multiple years 

Fielding 

Mail CDs and 
instructions, or make 
available over Web 
site 

Concurrent with 
fielding of platform 

Fielding may occur 
over multiple years 

Technical 

and 

Engineering 

Data 

Source Code with 
comments. Database 
Dictionary 

Source Code with 
comments. Database 
Dictionary, interface 
documents 

Drawings and 
Specifications 

Training and 
Manuals 

Required and updated 
about annually with 
each version 

Required 

Required 

Operations 

and 

Maintenance 

Intense, with 
revisions required 
about annually 

Concurrent with 
periodic platform 
maintenance 

Periodic 


PDeveloped by Researcher] 

Table 5. Comparing Management Information Systems to Other Systems 
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Testing for Management Information Systems can often take place in an office 
environment. Through beta testing, users provide feedback on the initial system 
functionality and operations. 

Production for software is simple. Once the version testing is complete, 
production may include printing Compact Disks (CDs) with the installation code loaded. 
Changes to Management Information Systems are relatively easy to field. Fielding often 
involves the relatively simple process of printing and mailing CDs with installation 
instructions. Unlike hardware, software changes are easy to code and field, as well. The 
limitation on system upgrades comes primarily from the areas of testing and 
documentation. These processes are more time-consuming and costly for software than 
for hardware. Additionally, the technical and engineering data are different for software 
and hardware. 

For Management Information Systems, there is a greater flexibility in the areas of 
training and manual generation, revision, and support. Training can take place on a 
computer system with only a simple interface, and for embedded software or hardware, 
modeling and simulation can provide a realistic training environment. However, tr aining 
is not complete without spending time on the actual hardware system. 

The majority of activities involved in the O&M of software requires technical 
expertise, experience, and directed knowledge of the system architecture and code. Data 
administration and software changes introduce the most risk when performed by less 
skilled personnel. There are several levels to perform maintenance of most hardware 
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systems, and the maintainers can more readily understand the system through technical 
drawings and other documentation. 

2. Department of Defense Requirements for Software Metrics 

The DoD and its Services have been in the forefront of automation and software 
development since the first computer was developed. For example, the DoD sponsored 
the early development of supercomputers to calculate ballistics, the computer language 
ADA, and the Oracle database. In 1992, DoD recommended the four Basic Measures for 
software systems shown in Table 6. 


Unit Of Measure 

Characteristics Addressed 

Counts of physical source lines of code 

Size, progress, reuse 

Counts of staff hours expended 

Effort, cost, resource, allocations 

Calendar dates 

Schedule 

Counts of software problems and defects 

Quality, readiness for delivery, improvement 
trends 

[After Ref 2] 


Table 6. DoD Recommended Measures from 1992 


The DoD requirements for software measures established in this 1992 document 
did not change significantly when addressed again in 1996. In a 1996 policy memo [Ref. 
17], OSD requires that, at a minimum, software metrics should address the following six 
areas: 
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1. Schedule and progress regarding work completion, 

2. Growth and stability regarding delivery of the required capability, 

3. Funding and personnel resources regarding the work to be 
performed, 

4. Product quality regarding the delivered products, 

5. Software development performance regarding the capabilities to 
meet program needs, and 

6. Technical adequacy regarding software reuse, ADA, and use of 
standard data elements 

The U.S. Army has developed a program called the Software Test and Evaluation 
Panel (STEP) to address software metrics and their use. DA Pamphlet 73-7, dated 25 
July 1997 [Ref. 10] provides the published list of Army Software Metrics. Table 7 
summarizes the list of metrics included in the document. The Air Force also 
recommends the Army STEP Metrics for program management of software [Ref. 9]. 

The STEP Metrics use many standard software metrics, adopting best commercial 
practices. These metrics are inclusive of fielded systems, including measures of mean 
down times, fault profiles, and reliability. The three categories for metrics are 
management, requirements, and quality. 
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QUALITY CATEGORY REQUIREMENTS MANAGEMENT 

_ CATEGORY CATEGORY 


METRICS 


OBJECTIVES 


MEASUREMENTS 


Cost 

Schedule 

Computer resource 
utilization 

Software engineering 
enviromnent 

Track software 
expenditures 

Track schedule adherence 

Track planned and actual 
resource use 

Quantify developer 
software engineering 
environment maturity 

Dollars spent vs. dollars 
allocated 

Milestone/event slippage 

Percent resource capacity 
utilized 

Computed maturity level 

Requirements traceability 
Requirements stability 

Track requirements down 
to code 

Track changes to 
requirements 

Percent requirements 
traced 

Number of requirements 
changes 

Design stability 
Complexity 

Breadth of testing 

Depth of testing 

Fault profiles 

Reliability 

Track design changes 
Assess code quality 

Track testing of 
reqmrements 

Track testing of code 

Track open vs. closed 
anomalies 

Assess software mission 
failures 

Measure down time 

Stability index 

Complexity indices 

Percent requirements 
tested 

Percent requirements 
passed 

Degree of code testing 

Number and types of 
faults, 

average open age 

Mean time between 
failmes 

Restoration times 


[From Ref. 16] 

Table 7. Army Software Test and Evaluation Panel Metrics 
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Management metrics address the basic areas of cost and schedule. Further, the 
maturity level is an assessment of the processes used by the software organization, in 
accordance with the SEI SW-CMM. Within the requirements category, traceability and 
stability establish quantitative values for the requirements generation and change 
processes. The quality of the system provides a series of measures that help managers to 
assess how well the system functions through changes, complexity, testing, and 
measurement of failures. 

3. Commercial Software Metrics 

Research shows that published metrics for software systems support the 
development phase, without specifically addressing support of fielded systems. 
Commercial software metrics do not introduce additional metrics beyond those used for 
DoD. Therefore, the set of software metrics established for DoD and the Army is 
inclusive of the typical measures and tools that commercial industry uses for management 
of software programs. 

B. FIELDED SYSTEM METRICS 

Fielded systems measurement should address issues related to continuous 
management of the system, and issues relating to the use of the system. These metrics 
include those that come directly from the system customers. The list of customers 
includes upper management, those involved in funding, and those who perform data 
input, those who access system data, and the system maintainers and administrators. 
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Aspects of use that merit consideration for metrics are user satisfaction, response time, 
failure rates, equipment, spares and replacements, and quality problems. 

The measures of fielded systems should provide specific outcomes. Cost drivers 
related to system operations are key areas that warrant measurement. Cost drivers 
include user help and training, maintenance of software, (e.g., correction of defects, 
enhancements), hardware and software purchases, and data maintenance and 
administration. These cost drivers can help to develop metrics that continuously 
diagnose problems and assess the outcomes of corrective actions. 

1. Department of Defense Logistics Metrics 

The DoD establishes metrics for its fielded systems based on established criteria. 
The basic goals with fielded systems are to maintain system readiness through 
maximizing availability and minimizing maintenance factors. 

The DoD environment differs significantly from the commercial enviromnent, 
because systems are in use for much longer in the military environment than in the 
commercial environment. These aging systems require more frequent maintenance 
activities. Maintenance on these older systems is more difficult, requiring better records 
and repair parts. The short life of commercial automation hardware and software, and the 
associated support cycle are not consistent with the military concept. 

Table 8 provides a list of the Objectives and Performance Measures that are 
included in the DoD FY 2000 Logistic Strategic Plan [Ref 7]. These measures are part 
of the overall logistic plan for all fielded systems within DoD. Readiness is a primary 
concern for logistics, and most of these DoD measures relate to readiness. In general, the 
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DoD is concerned with having systems that are available and perform well, while 
controlling costs. 

The objective to improve strategic mobility to meet Warfighter requirements is to 
help achieve readiness from mobility of assets. Mobility is only an issue for 
Management Information Systems in the rare case where a site needs a node, 
workstation, and connectivity, urgently. Having assets available to address these urgent 
requirements can satisfy another objective, to fully implement joint Total Asset Visibility 
(TAV) across DoD. Knowing the location, configuration, and usage of workstations and 
user accounts is the key to meeting this objective for Management Information Systems. 


1 OBJECTIVE 

MEASURES 

Optimize support 
to the Warfighter 

Measure: Attain the specified percentage of Level “A” weapon 
systems meeting their targeted Mission Capable (MC) Rate through 

FY 2006. Develop a documented baseline of applicable Military 
Service MC rates by the end of FY 2001. Establish target MC rates 
for the end of FY 2006, and track progress in attaining these targets 
(i.e., the percentage of increased/decreased MC rates) annually 
beginning at the end of FY 2001. The Military Services will develop 
the capability for quantifying the actual and target MC rates based on 
individual weapon systems, weapon system categories or Service 
composite, as appropriate, to provide meaningful performance 
information. The Defense Logistics Agency will develop the 
capability for reporting its Customer Wait Time (CWT) baseline and 
annual progress toward the Defense Logistics Agency FY 2006 CWT 
target, by Service or consistent with the Services’ weapon system 
categories. Each Component will report its composite progress 
against its target(s) annually, beginning at the end of FY 2001. 

Improve strategic 
mobility to meet 
Warfighter 
requirements 

Measure 1: By the end of FY 2006, achieve a cargo airlift capacity 
and sealift surge and afloat pre-position capacity to meet the validated 
requirements in the current Mobility Requirements Study. 

Measure 2: Develop a measurement plan and goals for mobility 
infrastructure and mobility process improvements by the end of FY 

2001. Achieve those goals by the end of FY 2006. 
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Implement 
Customer Wait 
Time (CWT) as 
the DoD logistics 
metric 

Measure 1: Develop the process for definition and measurement of 
Customer Wait Time (CWT) by the end of FY 2001. 

Measure 2: Fully implement CWT measurement for 100 percent of 
all selected segments by the end of FY 2006. 

Fully implement 
joint Total Asset 
Visibility (TAV) 
across DoD 

Measure: Determine user/business methods, asset information 
requirements and associated measures by the end of FY 2000, 
implement 100 percent of requirements by the end of FY 2006. 

Reengineer/ 

modernize 

applicable 

logistics 

processes/ 

systems 

Measure: Develop Component logistics processes/systems 
modernization plans by the end of FY 2001, and increase the 
proportion of modernized logistics business systems according to 
those plans by the end of FY 2006. The Military Services, the 

1 Defense Logistics Agency, and the U.S. Transportation Command 
will develop the capability for quantifying the percentage of logistics 
and related Automated Data Processing (ADP) systems modernized 
using implementation status as of the end of FY 1999 as a baseline. 

For each major system undergoing modernization, track annual 
progress against an FY 2006 target percentage. Each Component will 
report its composite progress against its target annually beginning at 
the end of FY 2001. 

Minimize 
logistics costs 
while meeting 
Warfighter 
requirements 

Measure: For selected fielded weapon systems, reduce the logistics 
support cost per weapon system per year compared to FY 1997 
baseline as follows: 7 percent by FY 2000; 10 percent by FY 2001; 
and a stretch target of 20 percent by the end of F Y 2005. 


[After Ref. 7] 

Table 8. DoD FY 2000 Logistic Measures 


These measures, identified in Table 8, correlate to Management Information 
Systems, two of which are strongly related to system readiness and availability. These 
are: 1) Optimize support to the Warfighter and 2) Implement Customer Wait Time 
(CWT) as the DoD logistics metric. For Management Information Systems, availability 
and readiness are measured in terms of system down time, to include server, workstation. 
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and connectivity. Measuring mission capability for Management Information Systems 
also includes system down time and the processing speed for the system. 

To reengineer and modernize applicable logistics processes/systems is to establish 
and maintain a system of goals and measurements for the Management Information 
System. Programs can achieve this through development and periodic assessment of 
business plans, schedules, objectives, and measures. 

O&M funds are the typical type of funding used for logistics costs, and 0«&:M 
dollars are a scarce resource within DoD. Once a system is fielded, the manager’s 
primary objective is to minimize logistics costs while meeting Warfighter requirements. 
The logistics costs for fielded Management Information Systems includes maintenance of 
all user requirements, upkeep of the system, and iterative maintenance upgrades. 
Revalidating the validity of requirements helps to control costs. Another cost control 
measure for logistics is consideration of the system architecture. Application of new 
technologies emerging from the IT industry can also help reduce logistics costs to fielded 
software systems. 

2. Industry Measures for Fielded Systems 

The basis for most measures that industry uses for fielded software systems is the 
ability to earn profit on sales of future end items or components, to improve return on 
investment. Industry wants to keep the customer satisfied, if the satisfaction can lead to 
future purchases of the end item or related items. In commercial industry, companies 
also want to measure satisfaction in an effort to improve the next product version enough 
to increase customer loyalty and, consequently, improve future sales. 
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In order to measure and improve customer satisfaction for fielded software 
systems, it is essential to understand what factors actually drive customer satisfaction. 
These drivers determine attributes to measure and functions or processes to improve. The 
concept of customer satisfaction in industry determines future customers for the product 
or service. In DoD, most users do not have the choice to “shop around.” The NICP and 
Treaty community have selected CAMIN as the system that the sites must use. 
Nevertheless, assessing and improving customer satisfaction is beneficial to mavimiVin o 
efficient and timely use of the system by existing users. Customer satisfaction is a broad 
area, encompassing these four dimensions of program management: system performance, 
customer expectations, customer perception of his or her interaction with the PMO, and 
problem resolution. 

C. CASE INTERVIEWS 

Comparing other similar fielded software systems to CAMIN yields insight into 
the program management measures that managers typically use. The following two DoD 
systems have characteristics in common with the CAMIN System. These systems are the 
Engineering Data Management (EDM) System and the Federal Emergency Management 
Information System (FEMIS). An analysis of these systems provides information 
concerning the systems, including a description, their unique parameters, and the metrics 
used in program management of these systems. 
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1. Engineering Data Management System 

(Ref. 1) 

a. System Purpose 

The Engineering Data Management (EDM) system O&M resides within 
SBCCOM to maintain and provide access to all technical data for the command 
Research, Development, and Acquisition Center. The purpose of the EDM system is to 
manage SBCCOM engineering data. EDM is a Product Data Management (PDM) 
system, and PDM is the commercial name for the generic family of products to manage 
engineering data. Data includes the engineering drawings and specifications and the 
relationships among those documents to include configuration management relationships. 
The system controls data entry, data access, and associated user permissions. 

b. System Description 

The EDM is both thin client and a client-server system. The user interface 
is completely based on thin client architecture. Thin client means that no software code 
or data are stored on the client workstations. The client-server system predates the thin 
client, to reduce operating cost and complexity. Once thin client is available for all user 
needs, the client-server system serves only system administration and data administration 
fimctions. 

For a thin client system, the workstation requirements are very basic. 
These COTS software packages are required: a 128-bit encrypted web browser, Adobe 
Acrobat Reader, WordPad or Unix text reader, and a raster reader. The imager for the 
raster reader is available free from the Army. The PDM COTS product used in EDM 
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performs these functions. It manages interfaces and data access for the Graphical User 
Interface (GUI) system administration tool. It manages the vault and the storage media 
where images are located. Finally, it manages the Oracle database that retains the Meta 
data for the documents stored in the system. 

The EDM uses three different types of thin client interfaces. First, the 
COTS thin client interface provides all the same capabilities as the full client system, 
with the exception of system administration tools. Permission and group controls permit 
operation of the interface. Second, EDM has a custom interface that provides a simplistic 
view and access to the system. The user-based capabilities of the web interface are the 
same as the full system, but the limited features simplify navigation wi thin the system, 
and users find this interface easier to leam and use. Systemic user problems drive the 
need for this interface. When users are not accessing the system fi-equently, the users fail 
to gain the familiarity required to use the more complex version. The third system access 
portal is another custom interface. This interface is very specialized, providing access to 
selected users for assignment of drawing numbers. Drawing numbers are an index field 
in the system database, and the controls must exist for assignment of these numbers. 

EDM also has a workflow management tool as part of the PDM COTS 
package. The workflow management tool automates the engineering data processes. It 
tracks the processing and status of the necessary events to generate and release new data. 

The user community for EDM is limited to DoD employees and 
contractors that require access to the data. The acquisition community uses the system to 
provide the baseline for procurement of end items and of spare and repair parts. The 
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engineering community uses it as a repository for designs, reference source for design 
improvements and new developments. Engineers also use the system to provide 
information in support of investigations of field failures. In the current configuration, the 
system is simple enough that no user manuals or formal training of the general user group 
is required. The EDM PM operates a help desk, and provides straightforward Avritten 
user procedures, upon request. 

c. System Metrics 

The PM for the EDM has no formal program for metrics, but uses the 
parameters described in this section for measuring system success. EDM measures and 
assesses customer response. The PM collects these data regarding the satisfaction of 
system users, mainly through interpretation of help desk calls. The help desk calls may 
indicate a systemic problem, operational problem, or training problem. 

Another measure is performance against operational requirements. 
Services provided are continuously improved based on the PM vision of what a PDM 
system should do in the current environment and what tools are available to do them. A 
group of PMs is performing similar tasks fi-om other AMC Major Subordinate 
Commands (MSCs). The PM for this system uses output firom the group to codify the 
vision. The result is a performance specification for the Automated Configuration 
Management System (ACMS) [Ref. 1]. 

In the area of resource management, EDM assesses costs and bill-payers. 
The EDM PM must budget well in advance of the actual need. During each year, the PM 
uses previously budgeted funding to contractor consultant hours to fix bugs and 
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implement needed enhancements. Historical data helps to estimate needed funding. 
Annual operating cost is approximately 20 percent of the initial cost to field an 
operational system. This figure correlates to the annual maintenance fees charged by 
most COTS manufacturers. 

The EDM has two types of paying customers. Project teams that are 
requiring engineering data to be stored in the repository pay variable costs. The 
SBCCOM RDA Enterprise pays the fixed costs, so that the engineering and acquisition 
community can use the system to access the data. These customers are widely satisfied 
with the system, and budget to maintain the system at required levels. 

Quality is another concern of the EDM program manager. The PMO tests 
each software change, and performs troubleshooting when a problem is suspected. They 
maintain a test bed to facilitate regular testing. Testing methods consist mostly of testing 
various Use Cases and reporting problems back to software consultants for correction. 

2. Federal Emergency Management Information System 

(Ref 12 and 13) 

a. System Purpose 

The SBCCOM and the Federal Emergency Management Agency jointly 
developed the FEMIS. The FEMIS is an integrated system that provides pl anning 
coordination, response, and exercise support for emergency management personnel. 

During an emergency response, the FEMIS users retrieve and execute 
plans created under non-emergency conditions. FEMIS uses real-time data from outside 
sources, such as meteorological monitors. To support the decision-making process. 
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FEMIS provides these outputs to the user: resource tracking, task lists, contact lists, event 
logs, status boards, hazard modeling, and evacuation modeling. 

b. System Description 

Local, state, and Federal emergency management experts developed the 
requirements for FEMIS. Users were actively involved in the prototype reviews, testing, 
and setting priorities. Continuing development is the result of user input and review. 
FEMIS meets Federal and industry open systems standards. The system architecture 
consists of a suite of COTS software products, including ArcView, Oracle, and Microsoft 
Project. The U.S. Government provides the hazard analysis and evacuation models. 

The system platform is client-server based to support multiple users, 
distributed data, and multiple operations centers. FEMIS uses a UNIX central data server 
and personal computer clients with Windows NT. The size of the FEMIS is 
approximately 1.25 million SLOCs. The system has up to 350 users. Leased lines 
coimect system nodes. The system uses a LAN connection from client to server. 

The FEMIS system development is recent. Requirements definition 
occurred in 1993. The first system fielding occurred in 1995 as version 1.1. General 
system fielding occurred in 1999 as version 1.4.5. The current version is 1.4.7. 
Maintenance releases, with enhancements, occur about every 8-12 months. 

The PM for FEMIS is evaluating the conversion of the user interface to a 
thin client structure. A conversion would enhance the FEMIS, reducing firewall issues, 
improving system accessibility, and saving life cycle costs. 
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c. System Metrics 

The FEMIS PM uses the Software Enhancement and Problem Report 
(SEPR) database, maintained by the contractor, to measure the system. The reports track 
SEPRs by three aspects: severity, time to close, and rate in versus rate closed. The SEPR 
process introduces all enhancements to the system. The PM uses the following two 
measures for tracking SEPRs: severity measure, which is commercial practice, and 
measure of DoD priority. 

D. PDSS CANDIDATE METRICS 

This section provides descriptions of the candidate metrics for fielded software 
systems. Table 9 summarizes the metrics from this chapter, and cross-levels among the 
possible sources of metrics, to include: 1) software metrics, 2) fielded system metrics, 3) 
metrics derived from the case analysis, and 4) CAMIN metrics. The metrics collected 
vary in scale, and major metrics encompass minor metrics. The combination of metrics 
from these various sources enriches the definition of the metrics with respect to reality in 
a PDSS environment. This section also discusses each metric and its usefulness. Table 
10, presented near the end of this chapter, encapsulates the detailed list of PDSS 
candidate metrics and the associated measures, derived from of this research. 

1. Management and Planning 

Measuring and tracking costs and schedule are important to ensure that a program 
remains within existing funding and meets schedule requirements. These measures help 
the PM to predict cost and schedule of the fielded software systems for future years, and 
to help identify areas for cost control. The funding rates identify problem areas and 
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opportunities for diversion of funding from lagging areas to more urgent and important 
ones. Counting hours expended is a measure of trends and assessment of efficiency. 
Software design and development is an art as well as a science, and writing new code 
may not adhere to a strict schedule. As a result, slips in schedule may affect available 
funding. 

The PM uses measures to assess the cost expenditures for software tasks, 
compared to the initial cost estimates. Program cost measures assess trends and 
improvements, and provide data for future cost estimates. PMs need to track obligated 
and expended costs against progress and number of defects. Charts and graphs of these 
data improve the data analysis process. Schedule charts that managers can use to 
measure progress include Gantt and PERT charts. 

The Constructive Cost Model (CoCoMo II) [Ref 6] is one method for overall 
management when considering cost, schedule, and performance. CoCoMo II provides a 
range on its cost, effort, and schedule estimates, from best case to most likely to worst- 
case outcomes. 

It is important to remember that software programs are notorious for going 
beyond schedule and over budget. Planning provides a systemic approach to controlling 
schedule and other program problems. A strategic plan for the system with objectives, 
goals, and measures becomes a strong baseline for achieving control. 
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Soft^vare Metrics MetriS^^™ CAMIN Metrics 


Cost and Schedule, 

j Software Cost, Planning 

Engineering Levels 

Environment 


Computer Resource 
2 Utilization, 
Manpower, Effort 


Requirements 
- Traceability and 
Stability, Fault 
Profiles 


Design Stability, 
Complexity, Size, 
Progress, Breadth 
and Depth of 
Testing 


Cost, Quality 



Funding Obligated 
and Disbursed, 
Schedule, Program 
Management, 
Quality Assurance 


Critical Resource 


Requirements 

Validation 


Rate in versus rate „ . 

closed, Severity, PCR, 

Time to Close, DoD Configuration 
priority Management 



Fault testing 


5 Reliability 


System Readiness 
and Availability 




Problem Resolution 


System Performance System Performance 
against requirements 


Development, 
Percent Completion, 
Peer Review 


User Accounts and 
Logins, Help, 
Training, User, 
Hardware, Software, 
Training, 
Maintenance 


Data Interventions, 
Actions, Intergroup 
Coordination 


Customer 
Satisfaction, 
Expectations, 
Perception of PM 


Customer Response 
and Satisfaction 



[Developed by Researcher] 

Table 9. Candidate PDSS Metrics 
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The Software Engineering Environment (SEE) metric is a measure that is oriented 
toward software development, but has application for fielded software systems. This 
measure rates the developer's application of software engineering principles. Examples 
of these principles are the use of structured design techniques, the extent of tool usage, 
and the use of requirement management techniques. A qualified independent group 
should perform rating of software developers. One example of this is the SW-CMM 
[Ref 20] developed by the SEI at Camegie-Mellon University. 

The results of auditing software quality assurance provide an estimation of 
product quality and compliance of staff to the internal processes. A review of results 
provides the status of action items from life-cycle reviews. Trouble reports provide 
insight into the quality of the product and processes, and the effectiveness of testing. 
Peer review results provide insight into the quality of the intermediate and final products 
and into the peer review and development processes. Defect prevention provides insight 
into the cause of defects and the effect of defect prevention activities on defect insertion 
rates. Quality assurance is a measure of how compliant the program staff is perfo rming 
activities associated with the SEE. 

2. Resource Utilization 

i 

Resources utilized for a program include computer resources, equipment, and 
staff. It is beneficial to maintain utilization of these resources at a constant level. 
Measurement of these resources over time provides indicators of fluctuation of activity 
and inefficient operation. 
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Computer resource utilization (CRU) measures performance with respect to its 
computer resource utilization goals and requirements. Measures of computer resource 
utilization provide another tool for measuring trends in costs. This measure monitors 
utilization of development resources. 

The staff and their equipment are the critical resources of the fielded software 
systems. The PM can optimize costs, expenditures, and schedule through awareness and 
management of these resources. Measure of effort provides visibility into the 
contribution that staffing has on project costs, schedule adherence, and product quality. 
This measure helps the PM to assess current cost expenditures and anticipate future costs. 
This metric provides an indication of the developer's application of human resources to 
the development program and the ability to maintain sufficient staffing to complete the 
project. The manpower metric is composed of two parts: an effort measure monitors 
labor hours planned and expended, while a staffing measure accounts for quantity and 
types of personnel needed to do the work. This metric assists the Government in 
determining whether the developer has scheduled a sufficient number of employees to 
produce the product in the time and budget allotted. 

3. Requirements and Configuration Management 

This metric provides for the assessment of requirements traceability, validation, 
evaluation, tracking, and implementation. Changes to requirements are inevitable, and 
drive up the life cycle cost of fielded software systems. Maintaining comprehensive 
measures of requirements changes helps to control costs and maintain stability. 


66 




The requirements traceability metric measures the portion of the software system 
that has implemented the documented system reqmrements. Also measured, is how well 
the system requirements align with requirements in higher-level docvunents. The 
software system includes software code and associated items, including specifications, 
software design, code, and test cases. Tracking System Requirements provides an 
historical record. 

Configuration management for fielded software systems includes submission and 
processing of change requests. Change requests are an important source of user 
requirements. Measures associated with configuration management help to assess trends 
in requirements and progress to accomplish these requirements. Users, funding 
managers, or data customers typically establish priorities and severity/importance for 
change requests during the CCB. When users present change requests, the response time 
may be important. Changes to CAMIN through Version updates can take up to two years 
fi-om contract task to fielding. 

4. Development and Maintenance Upgrades 

The degree of difficulty of development and maintenance upgrades is dependent 
on the size, complexity, and stability of the software architecture and code. This metric 
addresses the entire process to develop and modify software, including design, code, test, 
and field. 

Software that is more complex is harder to understand, test adequately, and 
maintain. Additionally, a highly complex unit has a greater tendency to contain 
embedded errors than a unit of lower complexity. The likelihood of introducing errors 
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when making code changes is higher in complex units. The design stability measure 
evaluates changes made to the design of the software. The complexity metric provides a 
means to measure and evaluate the structure of software units. 

Size measures are used to estimate cost, schedule, and workload. Size of software 
implies the scale of the system. Measures of size include source lines of code (SLOCs), 
Function Points, and Feature Points. A measure of the size of a system can vary 
depending on method of measure. The purpose for measuring size is to define the 
complexity of a system and how long it takes to write or change the system. One 
measure is to count the number of lines of code, called “source lines of code,” or SLOG. 
The software community accepts this method, especially for use with older programming 
languages. Newer programming languages, including those based on Object-Oriented 
logic, do not compare through lines of code. The only way to measure the size of a 
system for comparison purposes is to establish function point or feature point measures. 
Further, system size provides useful comparison only when measured consistently. It is 
important to be consistent in the method of counting to ensure common interpretation of 
the measure. The International Function Point User Group (IFPUG) [Ref 15] has 
established basic counting methods to provide consistency in the methodology. 
However, older systems do not typically use function point analysis, so comparison is 
impossible. The DoD continues to require SLOG as the standard measure of system size, 
whereas function and feature points include complexity ratings and other factors. 
Measuring size of software is very controversial due to technological advances in code 
implementation. The basis for controversy is comparing items of like size. Systems 
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based on object-oriented design principles are not comparable to systems designed using 
functional principles. In object-oriented systems, duplication within classes means that a 
system may reuse lines of code. A primary feature of Object-Oriented design is to reduce 
complexity, maximize reuse, and minimize problems associated with future maintenance 
and changes. 

Measuring progress is to assess the degree of certainty to complete the project on 
schedule. Progress is a measure of how well the project is performing with respect to its 
schedule. The development progress metric measures the degree of completeness of the 
software development effort, indicating the readiness to proceed to subsequent activities 
in software development. One should begin collecting data during software requirements 
analysis and continue throughout software development and fielded software systems. 

The schedule and progress regarding work completion is a key program 
management tool for a software development or modification project. Developers and 
the PMO should negotiate the schedule to arrive at a realistic plan. For initial 
development and new developers, the estimate for time is usually underestimated. For 
developers that have specific program experience, as those on CAMIN, estimates for time 
and cost are very close to execution. 

PMs assume program risk if the software development tasks are primarily driven 
by schedule. Results often include reduced quality, reduced documentation, and missed 
deadlines. When schedule becomes an issue for release of a software modification 
package for the CAMIN program, the list of tasks included in a software modification is 
reduced, rather than pushing the developers to meet an unrealistic schedule. 
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Development includes all of the iterative tasks involved in the design and writing 
of code. For the CAMIN program, the PM does not monitor to this level of detail. 
Instead, the PM monitors the processes used and certifications. The users assess the 
output of code during final acceptance testing before release of version update software. 

The CAMIN contractor performs use cases and regression analyses to measure 
defect rates and correct problems. Use cases are scenarios that depict how the system is 
used. Ideally, there would be at least one use case for every possible use of the system, 
and testing would find every defect. For complex systems like CAMIN, use case analysis 
is very time consuming and therefore expensive. 

The PM should run independent testing of the system. The tests should occur at 
each system modification, before the PM approves the software for release. For CAMIN, 
as a Management Information System with a relatively small user group, a user-based test 
occurs. The PMO prepares a test plan that includes evaluation of requirements from the 
^^tial tasking. The test plan includes scenarios that verify the change is in accordance 
with the task. Developers systematically fix and retest defects found during user testing. 
Occasionally, the PM and contractor may negotiate a low priority defect that would delay 
fielding into a later release. During testing for CAMIN modifications, the number of 
defects found is considered low, as it ranges from none at the least to two. Typically, 
user testing also yields the submission of 30-40 new change requests. 

The breadth of testing, also known as "black box" testing, evaluates the testing 
performed on the system from the user perspective. This metric is concerned with 
obtaining correct outputs because of prescribed inputs. No test can cover every possible 
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combination of variables involved in a Management Information System. Variables 
include interface, platform, and use case. 

The depth of testing, also known as "white box" testing, measures the amoimt of 
testing achieved on the software architecture. This metric focuses on the visibility into 
how the software is constructed. 

5. Availability 

This metric encompasses the factors that keep the system ready, operational, 
accessible, and user-fiiendly. Important components of this metric are user, system, and 
data administration, along with user support factors such as training and help line. The 
reliability metric tracks system failures caused by software and the time it takes to restore 
the system to its previous operating condition after these failures occur. User, system, 
and data administration are areas of CAMIN that the PM uses to control costs, ensure the 
system functions, satisfy customers, and protect system data. 

The user list for CAMIN frequently changes. Usage is the basis for hardware and 
software purchases, and seat licenses to use software. Security advises administrators to 
deactivate a user accoimt if the user is not using the system. Control of user accounts 
also saves administrative costs. Measures have identified CAMIN workstation users who 
only use capabilities that are available on the CAMIN web site. The PM revoked 
workstation accoimts imtil needed. These measures also provide usefiil data to justify 
adding new thin client capability. 
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Training provides users with the tools and techniques for using the CAMIN 
system. For the complex operations in CAMIN, users need to repeat training after long 
periods of system disuse. 

User help provides an instant tool for CAMIN users. As discussed with the case 
interviews in this chapter, measures of user help provide insights into problems with the 
system, with the user manual, and the training program. 

User Hardware/Software List keeps track of the amoimt and types of software and 
hardware in the inventory. The amount of hardware and software fielded drives up 
maintenance costs. The amount and configuration of these products affect system 
functionality. 

The system maintenance activities improve efficiency of system operations. For a 
complex system, these activities may be time-consuming. However, the actions 
addressed in this section are required for maintaining system operations and addressing 
customer issues. 

6. Problem Resolution 

The area of customer satisfaction is the in-depth exploration of customer 
problems. The goal is to create an environment where respondents feel free to report and 
describe, in detail, problems they have experienced. After learning about specific 
customer problems, it is essential to follow up and determine how well these problems 
were resolved. 

Problem resolution is crucial to maintaining customer satisfaction. Most 
customers recognize that occasional problems are unavoidable and even inevitable. 
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However, the method and attitude that the staff uses to respond to those problems is often 
the difference between a satisfied and unsatisfied customer. 

Good problem resolution can increase customer satisfaction beyond the level that 
existed before the problem occurred. Customers who report that a company has exceeded 
their expectations fi-equently cite quick, customer-oriented problem resolution as the 
source of their satisfaction. 

7. Performance against Requirements 

Measuring the performance of a system is necessary to determine customer 
satisfaction. Defined user requirements provide the benchmark for measuring system 
performance, and the requirement baseline is always in flux. System testing through 
IV&V and operational testing provide measures of performance. These tests must be 
repeated periodically to correspond to changes to the requirements, enviro nme nt^ or 
system. 


8. Customer Satisfaction 

Understanding customer expectations and then meeting or exceeding those 
expectations is fundamental to creating satisfaction. Customers become satisfied only 
when the system meets or exceeds expectations. It is important to recognize that 
expectations are not static. Performance that satisfies a customer today may not be 
sufficient to satisfy that same customer in tomorrow's environment. In addition, the user 
group is in a state of flux and users with different experiences may have different 
expectations. 
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A customer's perception of his or her interactions with the PMO is another key 
driver of customer satisfaction. Quite frequently, how customers feel about their 
treatment is more important than the underlying quality of the product or service. Poor 
treatment leaves a damaging and lasting impression that is difficult to overcome. 

The user expects a system that meets his or her performance requirements and is 
easy to use. In the case of the CAMIN system, the IV&V was conducted with the intent 
of measuring the system against the full program requirements. To maintain 
configuration management, the requirements database maintains old and new system 
requirements. New requirements come from approved PCRs and from program 
management decisions. At each version release, the PM and contractor jointly conduct 
extensive testing on parts of the system that may have been impacted. The user, 
however, develops and changes expectations of a system. Most users are accessing other 
systems that they naturally compare. The other systems may respond more quickly, 
making this system seem slow. The other system may provide a simpler interface, 
causing users to request the same short cuts and interfaces. The users may expect to 
work in a familiar environment, such as Microsoft Windows or web browser. Thus, 
using differing computer systems, users develop expectations about speed and ease of 
access or operation. 
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Metric 

Measures 

1 

Management 
and Planning 

Cost and 
Schedule, 

Funding 

Obligated and 
Disbursed, 
Schedule, SEE, 
Planning Levels, 
Quality Program 
Management, 
Quality 

Assurance 

Computed maturity level 

Measures quality of procedures against published procedures 
Quality Audits planned/conducted 

Nonconforming conditions found/closed/Open 

Monitor Strategic Plan 

Measure adequacy of funding 

Measure of costs by year, and by amount of requirement or 
flmction point 

Measure of funding obligations and dispersals vs. plan (Dollars 
spent vs. dollars allocated) 

Measure of hours expended vs. hours forecast and requirements 
Hours expended by day, month, task order/project 

Cost expended by month, task order/project 

Baseline (cost/hours/duration) vs. actual & progress (% 
complete) 

2 

Resource 

Utilization 

CRU, 

Manpower, 

Effort, 

Percent resource capacity utilized 

Measure of staff levels and rate of changes by function 

Assess funding and personnel resources vs. work to be 
performed 

Measure of Computer Resource Utilization (CRU) vs. 
requirement 

Staffing / Equipment resources needed over lifecycle 

3 

Requirements 

and 

Configuration 

Management 

Requirements 
Traceability and 
Stability, 
Validation, Fault 
Profiles, Rate in 
versus rate 
closed. Severity, 
Time to Close, 
DoD priority 
Requirements, 

PCR 

Measure of problems / defects over time and over versions 

Number of requirements changes 

Measure of priority/Measure of severity 

Measure of difficulty 

Importance of change vs. Rate Closed 

PCR Rate in vs. Rate Closed 

Requirements Identified (Percent requirements traced), fulfilled, 
outstanding, validated (for a release) 

Requirements by release, by configuration item, by category 
(new, baseline change, design change, implementation change) 
Changes per configuration item (volatility) 

Backup success. Backup resources used 

Hardware/ Software Environment Changes 

Number of files checked out/Number of files changed 

Number of users who checked out/changed a file 

Number of times, length of time each file was checked out 

Number of lines that changed per file (code only) 

Average number of changed lines (code only) 
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4 

Development 

and 

Maintenance 

Upgrades 

Design Stability, 
Complexity, 

Size, Progress, 
Breadth and 
Depth of Testing 
Fault testing. 
Percent 

Completion, Peer 
Review 

Cost to complete. Hours to complete. Milestone/event slippage 
Software size by version (SLOC), by tasking (function point) 
Periodically assess the development, documentation, peer 
review, and testing procedures (use SEE, where possible) 

Breadth, Depth of Testing, Degree of code testing 

Count of defects foimd during testing 

Stability and Complexity indices 

Percent requirements tested / passed 

Number and types of faults, average open age, restoration times 
Time to implement (baseline vs. actual) by software 
change/Configuration Item 

Defects/Problems identified, corrected, validated, time and effort 
to correct 

Peer Reviews planned/conducted 

5 

Availability and 
Usage 

Reliability, 

System 

Readiness, User 
Accounts and 
Logins, Help, 
Training, User, 
Hardware, 
Software, 

Training, 

Maintenance 

Time system is not available for planned and unplanned outages 
Monitor Continuity of Operations Plan 

User vs. system usage. New users, and deactivated users 

Trainee feedback on availability, quality of training 

Last user training vs. error frequency for user 

Categorize help calls to identify trends 

Measure frequency of user help vs. date of last training 

Measure of system and document usage 

Count and categories of maintenance issues 

Count and time to resolve non-conforming conditions 

Mean time between failures 

Effort/hours expended in staff familiarization (training) 

6 

Problem 
Resolution Data 
Interventions, 
Actions, 

Intergroup 

Coordination 

Measirre actions and resolutions, time spent, type of problems 
Measure rate of actions vs. time to close, based on priority 

Problem Calls logged (by category, by users, by action) 

Time/effort to service calls 

Open problem calls 

Data changes/interventions logged 

Time/effort to make data change 

7 

Performance 

against 

Requirements 

Compare system performance to requirements documents 

Measure system performance against defined requirements 

8 

Customer 

Satisfaction 

Fertorm surveys for user satisfaction, user meetings, help line, 
satisfaction with program management staff, overall satisfaction 


[After Table 7, Table 9, Appendix D] 

Table 10. PDSS Metrics and Measures 
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The concepts of customer satisfaction in industry determine future customers for 
the product or service. In DoD, most users do not have a choice to “shop around.” The 
NICP and Treaty community have selected CAMIN as the system that participating sites 
must use. Nevertheless, assessing and improving customer satisfaction is beneficial to 
encovurage users to utilize the system and user support systems, such as the help line and 
manuals, and maintain timely and correct system data. The users interact with the PMO 
through the help line, meetings, telephone calls, and e-mails. The result is that the degree 
of user satisfaction determines how often the user utilizes the staff resources. 

Most users present problems through the help line. The staff must then follow-up 
help requests to be sure that the user considers the problem closed, and is satisfied with 
the outcome. Data collected in the help line database can provide access to useful 
measures. It is also important to provide a mechanism for users to provide anonymous 
feedback on their perception of help provided. 

9. Final Metrics List 

The metrics and measures collected in this chapter provide a good baseline for 
analysis of systems in the PDSS phase. Table 10 provides a summary of the metrics and 
measures, consolidated by the eight categories that agree ■with Table 9. This list may 
prove useful in analysis of various PDSS that may be vastly different in design, 
application, or management. 
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IV. DATA ANALYSIS OF PDSS METRICS FOR CAMIN 


This chapter analyzes data collected in research for this thesis. The background 
presented in Chapter II contains a concise list of the CAMIN program management 
issues. Data from Chapter III generates a list of candidate metrics from DoD and 
commercial sources, and from other fielded software systems. Through analysis of these 
data, this chapter develops a list of the metrics that are recommended for use by the 
CAMIN program management. 

The analysis consists of evaluating key component areas for each issue from 
Chapter II, and identifying the parameters that require metrics and measures. Finally, this 
chapter analyzes lessons learned from a review of the CAMIN program, data collection, 
and the case studies contained in this thesis. 

A. SELECTION OF METRICS AND MEASURES FOR THE CAMIN 

This section analyzes the issues to select metrics for use by the CAMIN PMO. 
Table 11 smnmarizes these metrics. 

1. Funding 

The issue of funding ties into all of the other program management issues 
addressed in this thesis. Chapter II discusses problems that arise if the program funding 
is less than the requirement in a given year, and over the long term. Table 11 shows the 
metrics and measures that serve to address the programmatic flmding issues for CAMIN. 
Measuring success in funding includes other factors that help the PM to successfully 
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request and receive that funding. Following is a detailed discussion of each of these 
metrics, the measures, and methods of calculation. 

a. Understanding Funding Needs 

The PMO must evaluate each funding driver of the CAMIN program to 
determine if it is a valid need. The reason for this is that organizations responsible for 
funding the CAMIN system require revalidation of the need for requirements. All 
aspects of the program drive funding, so measuring all areas provides data to understand 
funding needs. 

The methods for evaluating funding drivers include the time and cost of 
implementing the new requirement, which may determine the validity of a requirement. 
If implementation is prohibitively expensive, the alternative to use a manual or other 
method may be preferred. However, there are intrinsic benefits to implementing new 
requirements that may not lend themselves to cost-benefit analysis. The PM must 
consider user need, mission accomplishment, data access and protection, productivity and 
retention of staff and resources, and the specific needs defined by the organizations that 
control the fimding when determining the validity of a new requirement. 

Another method for evaluating fimding drivers involves measuring the 
justification for funding needed to resolve PCRs, which comes fi-om the CCB evaluation 
of change requests. The CCB evaluation measures breadth of customer needfoenefit, cost 
and timefi-ame of meeting the requirement, and the direct benefit to system fimctionality, 
as well as the ability to meet the mission. The PM must retain CCB evaluation data to 
support future actions and to assess trends. 
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METRICS 


MEASURES 



Understanding Funding Needs 

Measure each funding driver for the 
priority/severity of need and for the 
likelihood of funding acceptance 

Funding 

Communication of Funding 
Justification 

Measure the degree of understanding by 
the target audience 


Receipt of Required Funding 

Measure and track the adequacy of 
funding in each area 


Power to the System 

Measure the reduction in system 
availability due to power failures 

System 

Connectivity 

Measure the consistency and effectiveness 
of cormectivity 

Availability 

Workstation/User Accessibility 

Measure the ability of users to access the 
CAMIN Server 


System Maintenance 

Measure the frequency, nature, and 
impacts of maintenance activities 


User Competency and 
Preparedness 

Measure PM actions that assist user 
competency 

Data and 

Outnut 

CAMIN Software Design 

Measure the design characteristics that 
cause CAMIN to meet requirements 

Accuracy 

Data Audit 

Measure the frequency and quality of 
audits to determine system data accuracy 


Data Interventions 

Measure the frequency, nature, method 
and time spent on data interventions 

Experienced IT 
Support 

Leveling of Tasks 

Measure the amount and types of work 
under the contract and the contract vehicle 

Challenges and State of the Art 

Measures the trends in technology 

Requirements 

Changes 

Changes to External 

Environment 

Measure the COTS software and hardware 
and the network environment to 
proactively adjust to changes 

Evaluation of Changes 

Measure the priority, type, complexity, 
and severity of change requests 


[Developed by Researcher] 

Table 11. Selected CAMIN Metrics 
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For requirements based in policy, regulation, or law, and for other 
customer requirements the PM determines the validity based on the source. It is 
important to determine if the requirement is required or simply advised, and under what 
circumstances the requirement is valid. When a paying customer submits a requirement 
with funding, the PM treats this requirement as valid and relatively urgent. The PM 
treats requirements introduced in direct support of the system mission to be valid and 
evaluated based on priority. 

For systematic maintenance requirements of the CAMIN system, the 
validation of the activities has a basis in historical data. Experience has provided a 
justification for the baseline maintenance activities that ensure consistent operation of the 
system. Experience also shows probable consequences if there is a lapse in maintenance. 

Measure: Measure each funding driver for the priority/severity of need 
and for the likelihood of funding acceptance. Examples of funding drivers are PCRs, 
upgrades to hardware and software, and software/hardware maintenance costs. 

Method of Calculation: Collect input from individuals that can 
adequately represent the user community and the mission requirements to evaluate the 
need. Measure the need on a predetermined scale (e.g. 1 to 10), to establish a baseline 
measurement. Evaluate the time and cost of implementing the new requirement. For the 
CAMIN program, the requirement baseline exists in terms of function point estimates. 
The combination of these measures provides the data needed to adequately validate the 
need. Knowing the cost, the need, and the benefit to the mission is sufficient to come to a 
business decision, based on the resources available. For example, the software/hardware 
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maintenance costs typically have a very high need value (10). If allowed to lapse for an 
amount of time, the cost would go back up to the initial purchase price. The benefit to 
the mission is assessed based on the potential situation if the need were not met. Without 
maintenance agreements, the program experiences high risk of not fimctioning, thus 
becoming incompatible with the external environment. The cost is estimated based on 
the purchase price fi-om the provider and the costs incurred by the contractor in the 
process. 

b. Communication of Funding Justification 

Another key element is to assess how effectively the PMO communicates 
its justification of the fiinding needs to fimd managers. This research identified no 
standard metric for communicating fimding needs for software or for hardware. 
Therefore, the methods of measure suggested here are new, and require evaluation. 
Measures of communication effectiveness should address communicating the right ideas 
about the funding requirements to the right people. 

Measurement in this area requires tools to assess how well the fund 
managers imderstand the funding intent and associated issues. The funding allocated 
may not be a useful measure of this, because other issues arise, such as availability of 
funding or emergency programs, which take precedence over the need for CAMIN. 
Nevertheless, a measure of funding requested vs. funding received provides a basis for 
the PM to study allocation of priorities and availability of funding. 

A valid measure may be available through direct feedback. The PM 
should continue to talk to fund managers throughout the funding process, and make 
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appropriate changes to the communication process, based on feedback received. 
Documentation of the communication process can aid in future communication of 
funding needs. 

Measure: Measure the degree of understanding by the target audience. 

Method of Calculation: Collect data from fund managers and peers on 
funding rates. Correlate the amount of funding requested to amount allocated. Evaluate 
data against other programs in the same funding pool. If the program fares unfavorably, 
pursue action to improve communication. Solicit periodic feedback from fund managers 
to assess their imderstanding of program funding requirements. Institute continuous 
improvement to mitigate problems when identified. 

c. Receipt of Required Funding 

Managing the program requires sufficient fimding to accomplish the 
activities required in support of the program. The program budget provides 
documentation of the funding requirements. In addition, the PM has flexibility to 
reallocate fimding within the program dollars to compensate for funding shortfalls. 
However, the PM must periodically assess the adequacy of the fimding in each area to 
use in planning for future years. One of the primary problems with reduced fimding in a 
given year is that higher fimding is required in the following year, to compensate. The 
next year, delayed maintenance causes higher maintenance costs, and changes to 
requirements become greater in magnitude, and therefore more expensive. Documenting 
the levels of fimding, and the effects of shortfalls on the requirements provides a record 


84 




that supports future funding needs. The documentation also creates a history to justify 
limitations in meeting system requirements. 

Measure: Measure and track the adequacy of funding in each area. 

Method of Calculation: Collect data on each new requirement and the 
extent of funding needed to meet that requirement. Note requirements not met for that 
year, and the effect on the current year. Include an assessment of the impact to the 
program in future years. In addition, document the higher maintenance cost each year, 
and jumps in technology caused by short funding in prior years. 

2. System Availability 

System availability is one of the fundamental system requirements. Lapses in 
system availability over the fielded lifetime of the system highlight the need for measures 
in this area. System availability encompasses user access to the system or data when 
needed. 

a. Power to the System 

As discussed in Chapter II, the constant supply of power to the server is 
not reliable. The CAMIN server is dependent on the infiiastructure wi thin the installation 
for its power. The mission requirement for system availability on a 24-7 basis is above 
the availability requirement for other computer systems on the installation. The onus is, 
therefore, on the PM for CAMIN to take the necessary action to meet its unique power 
requirements. 

Before taking action to prevent a power problem, the PM must collect data 
to determine the most cost-effective solution. One area to measure is the extent and 
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nature of power problems. The PM needs to measure the occurrence of power outages, 
the cause of those outages, and the frequency and duration. As the PM takes action to 
eliminate or mitigate power issues, documenting the trends is useful in justifying the 
actions taken. 

Another area to measure is the effectiveness of power protection systems 
that are currently in place. The systems to be tested include uninterruptible power 
supplies, the hot backup system associated with the high availability server, and the 
power backups used on the installation. The data collected from this analysis provides 
additional justification for enhancing the systems, if necessary. 

Another contributing factor to the importance of power interruptions, as a 
valid measurement tool, are impacts to the user community. For each power outage, 
users should be queried to determine the mission delay or failure during that period. 
Then, measure system usage rates over time to determine potential deleterious impacts 
from a theoretical power outage. 

Measure; Measure the reduction in system availability due to power 

failures. 


Method of Calculation; Keep track of the cause, the frequency, and 
duration of power outages. These data are available from the installation Department of 
Public Works. Query users about specific impacts to mission caused by power outages. 
Measure system usage over time through reports from the server. 
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b. Connectivity 

Connectivity problems can reduce user access to the system and data, as 
well. The PM is often unaware of connectivity issues imtil users identify problems 
because of inability to reach the C AMIN server. This forces the PM into a reactive rather 
than proactive mode. Maintaining connectivity may require maintenance modification of 
the system to remain compatible with the environment and network systems. Measures 
of connectivity involve three primary issues: 1) network, 2) firewall, and 3) security. 

The network is not available for significant periods due to local router 
problems, virus protective actions, and other reasons. Network connectivity is not critical 
to system operation, because the CAMIN system allows connectivity through either 
network or phone connection. However, the phone connection is relatively slow, 
compared to the network, inconveniencing users and delaying schedules. In addition, 
there are local policies that prevent phone connectivity via modem, and connection to the 
local network on the same workstation. The PM needs to track losses of network 
availability, and the causes, to identify systemic problems. The data is available through 
the local network administrator. 

The increased use of firewalls creates frequent connectivity problems. 
One or more firewalls exist at almost every CAMIN location, and each firewall 
configuration must allow CAMIN client-server communication. Network administrators 
frequently change firewall settings. A phone connection between client and server 
permits bypass of the firewall, if allowed. The PMO must remain in contact with local 
network administrators to track firewall configurations to assist when changes occur. 
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The firewall is only one of many security precautions that impede 
connectivity. Security is of primary concern to network administrators today. Other 
security precautions that impede connectivity include preventing ping and blocking 
access to specific user segments. The primary organization within the Army that 
manages these security issues within the military network is the Theater Network 
Operations and Seciuity Center (http://www.conus-tnosc.army.mil/) . a part of the United 
States Army Signal Command that is located at Ft. Huachuca, Arizona. The PM must 
collect and track information on current security activities to assure constant connectivity 
between workstations and the CAMIN server. For all of these connectivity issues, the 
primary goal of measurement is to be prepared to proactively manage problems. 

Measure: Measure the consistency and effectiveness of coiuiectivity. 

Method of Calculation: Examples of the measures for this aspect include 
monitoring and collecting data on firewalls, networks, and security changes. The PM 
should assess trends to identify changes before they occur. Data sources include all local 
network administrators and the U.S. Army Theater Network Operations and Security 
Center. Data output should be stored in a central database to identify trends. 

c. Workstation/User Accessibility 

CAMIN functionality is dependent on the ability of remote users to access 
the CAMIN server via the workstation. The previous section addresses network issues. 
Other issues that provide system access are proper configuration of the workstation and 
correct user permissions. 
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As Chapter II discusses, the workstation configuration is vital to CAMIN 
Client-Server functionality. Workstation configuration is difficult to assess and control at 
remote workstation sites. Published policies are limited in effectively managing the 
situation. However, the PM needs a method to assure that the workstation that is 
connecting to the CAMIN server is the correct configuration. The method currently used 
to meet this requirement addresses three aspects of workstation configuration. 

First, the PMO purchases, configures, and delivers the CAMIN 
workstations to user locations. This action provides initial control of configuration, but 
has limited use in guaranteeing future control. There is nothing to prevent users fi-om 
loading all CAMIN custom and COTS software onto a different workstation. 

Secondly, the PMO must guarantee that only PM-procured workstations 
connect to CAMIN. Version 8.0 of CAMIN software prevents connection to the server 
unless the identifying number of the workstation network card resides in the database. 
This action has been effective in maintaining control of workstations. 

Finally, the PM requires a method of obtaining data from remote 
workstations to assure compliance. The PM is currently implementing a system to collect 
workstation configuration data at remote sites. This new system provides users and 
workstation administrators with the capability to run a program, with output containing 
all required workstation parameters. The PMO can then request that the remote location 
run the program at selected intervals, and transmit the output data to the PMO for 
collection and analysis. 
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In addition, accessibility issue is user permissions. Users need to have all 
permissions that are required to perform needed work. CAMIN permissions restrict or 
permit access, by organization, by installation and facility, by application, and by action. 
The user administrator, currently residing in the PMO, sets these permissions. However, 
the need for specific permissions changes over time. The measure of permission against 
usage is key to establishing the correctness of these permissions on an ongoing basis. 
These data are available from the CAMIN server, and should be measured and evaluated 
for reallocation of permissions, quarterly. 

Measure: Measure the ability of users to access the CAMIN Server. 

Method of Calculation: The PM needs to collect and evaluate data on the 
configuration of CAMIN workstations to identify trends. The PM is currently developing 
a system to obtain these data. Where incompatibilities with the standard exist, the PM 
must enforce correction. The PM also needs to evaluate user permissions against needs 
and compare those permissions to actual system usage. This data is available on the 
CAMIN server. If users are found to have too many or too few permissions, the PM must 
take corrective action. 

d. System Maintenance 

System maintenance employs preventative actions to keep the system 
operational. Measuring system maintenance involves measuring the frequency, nature, 
and impacts of maintenance activities, and measuring the effectiveness of these activities 
(i.e. how often the system in unavailable due to lack of maintenance). Both sets of data 
are needed to conduct a complete trend analysis. 


90 




System maintenance activities include daily System Administration 
activities and implementation of system (software, hardware, and other equipment) 
upgrades. These actions must occur in a responsible manner, ensuring minimal impact to 
the user, while maintaining consistent system operations. 

Daily System Administration activities include management of file sizes, 
creating system and data backups, managing accounts and passwords, and performing 
periodic system testing to identify and correct problems. A daily log maintains 
documentation of these activities to ensure compliance and identify trends. 

Implementation of system upgrades can involve taking the system down 
for an extended period of up to a week. It is important to thoroughly test an upgrade 
before implementing it in a live user environment. The preliminary testing can anticipate 
most problems to precipitate preventative action, but unforeseen problems sometimes 
occur once the upgrade is in an operational environment. 

Taking the system down requires scheduling and coordination with system 
users and other customers. The PMO must build in flexibility to the schedule in case a 
short-notice treaty inspection occurs during planned down time. 

Measure; Measure the frequency, nature, and impacts of maintenance 

activities. 

Method of Calculation; The PM should measure all aspects of proactive 
and reactive maintenance activities to determine trends. The individuals that perform 
maintenance must log actions into a central database, and take corrective action if 
maintenance is not in accordance with prescribed levels. The PM should also track 
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events when the system in imavailable due to a lack of maintenance. The event-related 
data should include the frequency, duration, cause, and impact of the lapse. This data is 
available from system logs located on the central server. In cases where the system is 
unavailable, the PM should take corrective action. Systemic causes of failure wairant 
further action that may require changing the program strategy. 

3. Data and Output Accuracy 

As discussed in Chapter II, data and output accuracy are critical to the success of 
the CAMIN system. In order to achieve data and output accuracy, users must possess 
skill and knowledge when executing data input and output choices. The PM provides 

tools to prepare users and protect data. The measures assess the effectiveness of the 
tools. 

a. User Competency and Preparedness 

Measuring user competence is a political issue, tied to employee 
performance ratings and user support of the system. Direct assessment of user 
competence creates distrust and contention. As stated in Chapter II, CAMIN system 
users are sometimes not computer literate, and CAMIN data processing is often an extra 
duty included in a workday of disparate activities. The CAMIN PM has no direct control 
over the CAMIN users and their performance ratings. To gain user satisfaction and 
timely system use, the PM provides a service that the user can trust. The PM must 
balance the need for improved competency against the need for timely data inputs. 

Measures of user competency may include testing of users and/or 
punishment for user errors. However, the impracticality of using these direct measures 
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stems from the ability to enforce these measures, and from the resulting customer 
perceptions, should these measures be implemented. 

In addition, measures are not practical in all cases, as system users range 
from those that perform a wide range of user activities to those who only perform one or 
two basic actions. The test would have to be adapted for each user, eliminating the 
perceived “fairness” of equality of the measure. In other cases, testing may not be a good 
indicator for infrequent users. In the case of infrequent use, a call to the help line is the 
preferable way to provide instructions that link users back to their initial training. 

From the PM perspective, enforcement of a competency requirement 
means that those users that do not pass testing are removed from the user group. This 
action would be in direct conflict with the need for timely data input into the CAMIN 
system. Each site is short of staff, and there are CAMIN sites have no users that could 
pass the rigorous testing. 

Measuring and judging users has a significant impact on customer 
perceptions and associated trust. There is a concern that users, concerned for their job 
security, may hide data errors they created in the system to prevent punitive action. 
Another concern is that, rather than make a mistake, the user may do nothing, resulting in 
system data that becomes grossly out of date. The PM needs to maintain a relationship of 
trust with the users, and this type of judgment, on the part of the PM, creates divisiveness 
and contention. 

Rather than measuring and enforcing strict guidelines for user 
competence, the PM offers a selection of tools that the user can employ to improve 
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competency. These tools include training, access to help line support, and user manuals. 
Measuring the use of these tools does not measure user competence. However, 
measuring the frequency of use and the perceived benefits of use provides a measure of 
the tool and its utility to improve user competency and preparedness. 

Training for CAMIN users enhances data and output accuracy. Training 
provides instructions on system use, to include how to use the workstation, how to use 
applications, and the significance of data. Training also provides tips and exercises to 
improve the quality of learning. Users require retraining when the application changes 

significantly and when use of the system has lapsed for only a short period, typically one 
month. 

The help line provides special instructions and other help regarding system 
use and problems. The help line is a venue where users present problems about system 
function, use, and data. As a result, the help provided may include instructions, 
intervention, and/or PCR for system modification. Also, user manuals provide systematic 
instructions and diagrams for system users. Data and output accuracy can improve if the 
manuals are easy to use, informative, and factual. 

In addition to measiaring the tools provided to assist users, the PM should 
also keep track of system users and their abilities. Consolidation of the data collected for 
training and through the help line, data from user interventions, and data indicating 
frequency and duration of use by application, provide a profile of each user over time. 
This profile may be useful in establishing a baseline for competency, and in measuring 
improvement against that baseline. 
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Measure: Measure PMO actions to assist user competency. 

Method of Calculation: Direct measures of competence are beyond the 
CAMIN program control, but the PM does control the offering of materials and services 
to enhance user competence. A source of indirect measures is the assessment of training, 
the help line, and user manuals. Assessment includes how the PM provides these 
services, how users use the services, and the results of usage. Additionally, the PMO 
should cormt training offerings and attendance, and have instructors provide an 
assessment of trainee performance and improvement. If training is not resulting in user 
performance improvement, the PMO needs to redesign the training program. The PMO 
should also measure the frequency, nature, and duration of help line use. If trends 
indicate repeated problems, redesign of the system and/or manuals may be required. 
Also, assess the quality of help through trends in closeout rates and feedback from users. 
This user feedback can provide data to document the benefits of the manuals. The 
consolidation of data collected in this and other metrics provides a measure of user 
preparedness and an indicator of user competence. 

b. CAMIN Software Design 

The CAMIN software design elements should concentrate on data and 
output accuracy. Three considerations should be part of the software design: 1) ease of 
use, 2) data protections, and 3) continuous improvement. These three elements provide 
focus on the mission of the system to maintain accurate data, in contrast to a typical focus 
that supports the continuation of the existing design. 
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If the system is easy to use, users are more likely to use the system, and 
find it easier to use the system correctly. With a user base that includes infrequent users, 
a process-based system provides the benefit of reduced training requirements and more 
accurate system data. The potential downsides of process-based applications are reduced 
flexibility for more advanced users, and more time required to perform a given operation, 
(since users must go through a selection of all possible operations in the process of 
identifying the chosen one). Nevertheless, with data accuracy as a primary goal, the 
process-based approach to achieve ease of use is the prefeired alternative. This process- 
based design change would take place over an extended period, through a long-term plan. 
Feedback from users would indicate if the changes make the system easier to use. Other 
indicators of improvement are reductions in help line calls and in data errors. 

Another action that improves ease of use is the utilization of familiar 
P^^^forms and interfeces. Users perform more comfortably when working in femiliar 
surroundings. GAMIN currently operates on a Windows ™ NT Version 4.0 platform. 
When GAMIN converted from Windows ™ NT Version 3.51, an improvement in the 
comfort level of users and local administrators was noted. Future changes being 
considered for GAMIN include moving to a Windows ™ 2000 platform and/or to a web- 
based platform. 

In addition, system data protections reduce the chance for user error. 
When systemic data errors are identifled, an investigation of causes may indicate that the 
software can be redesigned to prevent the error or caution the user against maVing the 
error. If a redesign can help, the change is documented in a PGR and sent to the GGB for 
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disposition. Depending on the cost of the change, the assessed benefit, and the binding 
available, the change may be included in the next version of CAMIN software. The 
measure of success of the data protection action is the reduced occurrence of the 
identified data error. 

A process of continuous improvement of the design should work toward 
retention and improvement of data accuracy. One area of continuous improvement that 
may enhance data accuracy is the leveraging of the system application over several 
programs. Initially, the system was designed to address chemical treaty requirements 
only. By incorporating the wholesale army accountability mission, the data is reassessed 
through regularly scheduled audits and inventories. Additions of Materiel Assessment 
Review Board (MARB) data storage, CWPF accountability functionality, and the like, 
further enhance the data accuracy. Measurement of continuous improvement is indirect, 
but can be accomplished through reduction of data errors after improvements are 
adopted. 

Measure: Measure the design characteristics that cause CAMIN to meet 

requirements. 

Method of Calculation: Measure the ease of use of the system through 
user feedback. The PM should use formal and informal approaches to obtain user 
opinions. A measure of data errors by frequency, type, cause, and severity provides 
benefit, because it addresses the design areas. However, this measure requires high 
quality data audits, which are outside PM control. 
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c. Data Audit 

The data audit is the most systematic method available for assessing the 
accuracy of C AMIN data. Data audits must support each type of data within CAMIN, 
with representation from the pertinent mission areas. However, there are few formal 
procedures in place for audit of CAMIN data. Table 12 describes the organizations 
responsible for audit of identified CAMIN data. 

The key organizations involved in data are the NICP, the MARB, the 
treaty organization, the quality and maintenance staff within SMT, and local property 
accountability organizations. The NICP audits CW storage and destruction at the end of 
each destruction campaign. The NICP also requires the local custodial officer at the site 
to perform an annual inventory of all CW storage locations, reconciled against CAMIN 
data. 

When the site receives notification of a short-notice treaty inspection, the 
local treaty compliance officer may conduct an emergency data review. This effort may 
result in surface corrections, but does not replace the detailed review required to assure 
data is correct. During the treaty inspections, the international inspection team conducts 
a more comprehensive audit of CAMIN treaty data. However, inspections are 
undesirable times to discover data errors, requiring explanations and negotiations, and 
breeding distrust. 
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1 DATA 

1 CATEGORY 

ORGANIZATION 

DATA AREAS 


NICP 

Storage, Movement, and Destmction, 
Ownership Codes 


MARB 

MARB data 

CW 

Treaty 

Storage, Movement, and Destruction 


Quality and Maintenance 

Defect Codes, Condition Codes 


Local Property 

Accountability 

Storage, Movement, and Destruction, 
Hazardous Waste Info, 


Treaty 

Storage and Destruction 

Former CWPF 

Local Property 

Accountability 

Storage and Destruction 


Treaty 

Storage, Movement, and Consumption 

Schedule 1 

Local Property 

Accountability 

Storage, Movement, and Consumption 


Treaty 

Site Diagrams, Process Flow Diagrams, 
Installation and Facility Info 


NICP 

Installation and Facility Info 

Site 

Information 

Quality and Maintenance 

Planographs, Grid Definitions, 

Installation and Facility Info 


Local Property 

Accoxmtability 

Site Diagrams, Planographs, Process 

Flow Diagrams, Installation and 

Facility Info 


[Developed by Researcher] 

Table 12. CAMIN Data Audit Responsibilities 


None of the other audits listed in Table 12 take place on a systematic 
basis. This is a matter of concern in support of CAMIN data accuracy and associated 
measures. The PMO does not have sufficient staff to conduct these necessary audits, and 
the organizations with cognizance over the current data are better qualified to conduct the 
audits. 
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Data managers need to periodically audit the CAMIN database for 
accuracy. These audits help to identify unique systemic data problems. The data 
managers need to have an understanding of the data and output requirements. The 
CAMIN program needs data managers from both the Treaty and the wholesale 
accountability functions to work together in the interest of maintaining consistent data. 
Evaluation of data errors can help to determine if a design change can prevent future 
errors. Audit of data also provides a method for measuring the quality of system, 
training, and other user help systems. 

Measure: Measure the frequency and quality of audits to determine data 

accuracy. 

Method of Calculation: The PM should measure the frequency and 
output of audits, collecting data containing users, auditors, locations, and expected 
results. These data are available through queries of the organizations that should be 
conducting the audits. The PM does not have control over the audit process, but may 
provide feedback of this measure through the chain of command. 

d. Data Interventions 

Through data interventions, the Data Administrator corrects user errors, 
intercedes to bypass software design, and performs mass changes to system data. The 
actions are performed through manipulation of the data in the CAMIN database. Data 
interventions can be time consuming and add risk to data accuracy. The action does not 
leave an audit trail, so documentation of the intervention is important. The process for a 
data intervention includes design of the intervention, testing the intervention in the 
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training database, and implementation and notification of the users involved. A Data 
Administrator on the contractor staff performs all data interventions. The Data 
Administrator only performs data intervention with authorization of the PMO. 

It is inevitable that humans make errors. For many important data 
transactions, the CAMIN system requires that a second CAMIN user confirm the 
accxoracy of the transaction through the QA process. However, data errors continue to 
occur, even with QA. Before automation, when users made errors, they simply marked 
up the paper document. In the TCM, the system that immediately preceded CAMIN, 
users could enter the database and correct the basic data. Many errors in the database 
were perpetuated through these “back door” approaches, and CAMIN does not permit 
users to access or manipulate the CAMIN database. These data interventions for error 
correction improve the accuracy of data. Consequently, the software design has controls 
to prevent users firom violating laws, regulations, or treaty, and to protect the CAMIN 
data. In rare cases, there are exceptions to these rules, and the Data Administrator must 
intercede to bypass the software design. 

Infrequently, a change to policy, laws, regulations, or treaty requires a 
systemic modification to the CAMIN data. For these systemic data changes, the PM 
must determine if it is advantageous to the program for the Data Administrator to perform 
a mass change to the database, or if users must perform all transactions individually. 
Mass changes can improve system data through consistency of data input and through 
customer satisfaction. 
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Measure; Measure the frequency, nature, method, and time spent on data 

interventions. 

Method of Calculation: In all of these cases, the Data Administrator 
should document the process for each intervention so that it can be repeated. The Data 
Administrator should also record the frequency, nature, method, and time spent on data 
interventions. The PM can use the data to identify trends that may justify changes to the 
system design. 

4. Experienced IT Support 

The need for experienced IT support is fundamental to a successful program. 
Experience is important for two reasons: 1) The learning curves are very long for IT 
systems, and 2) Retention is difficult for IT professionals in the current economic 
environment, where demand for technical ability is high. The high performers assure 
career success by demanding high salaries and maintaining leading edge skills. 

This issue applies to both the Government and contractor staffs. Government 
personnel and salary systems provide challenges in hiring and maintainino these IT 
professionals. One reason is that the Government contracting system is oriented toward 
competition and controlling salary rates. As a result, the PM for CAMIN has no IT 
professionals on its staff. This creates a greater dependency on contractor IT experience 
to maintain the system. Introducing a competitive contract under the current situation 
would be extremely risky to the program. This also creates difficulty in the period 
between contracts, with no Government staff to provide continuity for maintenance. 
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a. 


Leveling of Tasks 


The IT tasks for the CAMIN system consist of system administration, d a t a 
administration, design, development, and test. This varied combination of IT work 
requires knowledge, skills, and abilities that rarely exist in a single person. Programs 
invest time and dollars in training and providing experience to a group of IT 
professionals. The program must have the funding and workload to support the retention 
of the group to perform the IT functions over a length of time that is sufficient to gain 
benefit from the investment. The minimum retention goal is three years. A reasonably 
constant amount of work in each of these areas supports the retention of the skilled key 
IT personnel. Level funding is one key element in maintaining constant workload. 
Another element that can assist fi-om a contract perspective is the type of contract. 

Level funding can occur through effective budgeting and responsible 
management. The type of contract and method of administration can improve the PM’s 
ability to retain an experienced IT support staff. The contract type should provide 
consistent and adequate funding, and work levels. Multiyear contracting can be helpful 
in leveling the tasks. The PM needs to periodically assess funding and contracting issues 
to determine the adequacy in maintaining a stable IT workforce. This data is available 
within the PMO and through contractor reports. 

The IT Fund, defined in U.S. Code, Title 40, Section 757, (Ref 21) 
provides valuable tools in addressing these IT issues for the contract. The Government 
Services Administration has authority to award and administer IT multi-agency contracts 
and to fund contracts on a reimbursable basis through a separate IT fund. GSA is the 
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only Executive Agent of that fund, designated by 0MB under the Clinger Cohen Act 
(Ref. 5). In order to promote the efficient management, coordination, operation, and 
utilization of resources; the fund has been established without fiscal year limitation, and 
allows for the use of multi-year contracts for IT hardware, software, and services. This 
multiyear contracting authority can be used when the following criteria are mete funds are 
available and adequate, the contract is fully competitive, the need continues for the 
contract period, it yields substantial cost savings, and it cannot exclude small businesses. 

The IT Fund provides very valuable tools, which can be used when contracting through 
GSA. 

Measure: Measure the amount and types of work under the contract and 
the contract vehicle. 

Method of Calculation; Periodically assess if funding and contracting are 
adequate to maintain a stable IT workforce. This data is available within the PMO and 
through contractor reports. If work is inadequate, the PM should provide data to decision 
makers to help make informed decisions about the program and funding. 

b. Challenges and State of the Art 

There are many good reasons for a program to keep up with state of the art 
technology, and maintaining experienced IT professionals on staff may be the least 
important of those reasons. However, keeping a challenging, “state of the art” program 
provides the program developers and other IT professions with another reason to remain 
on the program, as it provides a continuous learning environment. The technology on the 
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program should not stagnate to the point where schools and the rest of the IT world no 
longer use that technology. 

Measure: Measure the trends in technology. 

Method of Calculation: Assess the CAMIN system based on these trends. 
Obtain information on technology trends through trade journals, conferences, and 
contacts. Establish program strategies based on sound business practices, using these 
trends to support decisions. 

5. Requirements Changes 

Changes to system requirements are perpetual through the life of the system. The 
changes provide the continuation of system functionality in a changing environment. 
Frequency of changes drives the pace of the system and funding requirements. 

a. Changes to External Environment 

Changes to COTS software and hardware that are part of the CAMIN 
system drive changes to CAMIN. The reasons to adopt these COTS-driven requirements 
changes, include maintenance support availability and functionality within the external 
policies and environment. These factors are out of the control of the PM. The cost 
effectiveness of COTS is realized in the avoidance of development costs and the 
avoidance of maintenance personnel. Despite these factors, the Government and 
commercial industry have accepted this trade-off, and lise COTS as much as possible, as 
general policy. 

However, maintenance support is only available as long as the COTS 
developer chooses to support it. The IT industry is so volatile that support to a legacy 
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system often stops after about a year of fielding a newer version of that system. The 
CAMIN has flexibility even after software support has stopped. The COTS software will 
function as long as the interfaces still work, but may fail to comply with new policies. 
Once support stops on COTS hardware, the manufacturer does not offer maintenance 
contracts and replacements parts become difficult to fin d 

As a result, the CAMfM PMO must try to remain compliant with the 
fi-equently changing external environment and policies. Recent changes that fall under 
this category are networks and firewalls, Y2K compliance, and handicapped accessibility 
to web sites. These policy changes are often unpredictable, implemented on short notice, 
and imposed with no associated funding. The common link is that these external changes 
are beyond the control of the PM, and yet the PM must adapt to them. Program planning 
and fimding must allow for these kinds of changes. The PMO must maintflin awareness 
of environmental changes from COTS developers, and from Federal and industry policy 
organizations. 

Measure: Measure the COTS software, hardware, and the network 
environment to proactively adjust to changes. 

Method of Calculation: Track the history of changes to predict future 
needs. The PM can obtain information through Federal and trade journals, industry 
conferences, and contacts. Establish program strategies based on sound business 
practices, using these data to support decisions. 


106 



b. 


Evaluation of Changes 


The evaluation of changes to the CAMIN system takes place during CCB 
meetings. Voting membership in the meetings includes user representatives, funding 
representatives, and mission representatives. Consideration of priority, complexity and 
severity, and costs helps the CCB and PM make appropriate decisions, given resource 
limitations. As indicated, one consideration in evaluating changes is priority. Priorities 
are pivotal to decisions making, and often determining if the change is important enough 
to pursue. A determination of the type of change (defect, problem, or enhancement) also 
has influence on priority designation. The designations of priority and type are set by the 
user community and by the funding organizations, with input from the contractor. 

Fielding of changes to software can occur either through a full version 
release or through a patch. The contractor evaluates each change to determine its 
siiitability to be included in a patch. In addition, the size of the change determines 
whether to field the change through the NIPRNET or a compact disk. 

Another factor in the decision is the complexity and severity of changes. 
The contractor determines the complexity through function point analysis, and provides 
the data to the CCB. Managers may decide to postpone changes that consume large 
amounts of time and other resources, in order to accomplish actions that are more 
pressing. The cost factor for each PCR is significant, and is determined from the function 
point analysis, and the consumption of resources may make them prohibitively costly and 
time consuming to implement. 
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For implementation, the CCB collects all of these data to use in 
determining the disposition of change requests. The CCB analysis results in an action for 
each PCR that either approves, disapproves, leaves in an “under consideration” category, 
withdraws, or designates as overtaken by events. When funding becomes available, the 
PM considers approved and urgent actions in generating a contract modification to 
implement a version upgrade to the CAMIN software. 

Measure: Measure the priority, type, complexity, and severity of change 

requests. 

Method of Calculation: For each PCR, determine priority, type of 
change, suitability of the change to be fielded in a patch, complexity and severity, and 
cost. Data comes from the contractor, from CCB members, and from the submitter of the 
PCR. Data is stored in central database and used in tracking requirements and making 
upgrade decisions. 

B. LESSONS LEARNED 

The analysis from this thesis results in lessons that can provide benefit to the 
CAMIN PM. The lessons may also apply to other fielded software systems and IT 
systems managers, as well. These lessons learned are identified in the following 
paragraphs. 
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Ensure that decision makers fully understand the CAMIN program 
requirements (software, hardware, people, dollars). 

Decision makers must have the salient facts before making informed decisions 
about the program. Those making decisions about software, hardware, people, and 
dollars may not realize the significant differences between IT/ fielded software systems 
and other programs. For example, decision makers need to imderstand that updates of 
software are more fi-equent than hardware and persist through the life of the system. 
Further, each individual program has imique characteristics. The PM must keep decision 
makers informed on program issues and changes to the program. 

Ensure that users and their supervisors understand and take responsibility 
for having competent (trained and prepared) CAMIN system users. 

Users are essential to CAMIN and its data, and the users on the CAMIN system 
experience unique challenges of insufficient time and technology skills. Managers do not 
understand what is necessary in terms of time and money to select and maintain 
competent system users. Beyond that, many cannot afford to expend the necessary 
resources on training and maintenance of users. For example, managers and users may 
not realize that when use of the system is infrequent, users need to spend additional time 
on training and practice to maintain skills. 

Managers need to monitor the performance of CAMIN users under their 
supervision to assess the quality of performance. Insufficient audits take place at the 
oversight level to detect errors in data. Site responsibility for data is very important in 
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cases where only the site personnel know the “ground truth” about the data, and can 
detect data errors. 

Ensure that mission representatives understand the importance of data 

audits. 

Data audits can significantly improve the quality of data in CAMIN, and can help 
to monitor the skill level of system users. The lack of sufficient audits increases the 
program risk, by allowing existing data errors to go uncorrected. Those in a position to 
audit CAMIN data, as shown in Table 12, need to understand the importance of audits to 
the data relevant to particular programs. Performance of the data audits requires 
knowledgeable personnel, dedicating considerable time and effort to the action. The 
required frequency of data audits is variable, dependent on the volatility of data in the 
system. 

Maintain knowledge of trends and pending actions in the external IT 
environment, for both Government and industry. 

The environment has a significant influence on the CAMIN program and its 
resources. Changing client needs and changes to the external environmental drive 
requirements changes and subsequent system updates. External requirements for 
CAMIN can come from many sources, including the command CIO, AMC, Army, DoD, 
Congress, and the Treaty organizations. Changes come in the form of policy, regulation, 
or law, and are rarely accompanied with the funding required to implement. 
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Work toward Thin Client Architecture for CAMIN. 


For systems that have remote users. Thin Client is often more practical than 
Client-Server. The Case Interviews revealed a consistent conclusion with the Thin Client 
design. For the CAMIN system, each step toward total thin client, or web-based, 
architecture reduces the number of CAMIN workstations the program has to procure and 
support. Firewall issues become minimal. Access to data is greatly improved, as users 
with a CAMIN password can log into CAMIN from workstations with Internet 
connectivity. This action is consistent with the program objective to conform to the 
external environment. 

Achieve a constant workload for the CAMIN contractor to retain 
experienced IT support. 

Experienced IT support helps a program to perform more efficiently. Since the 
CAMIN program has no IT professionals in the PMO, there is a strong dependency on 
the contractor to provide continuity as well as experience. Performance becomes more 
efficient and consistent when leaning curves and experience curves are minimized. A 
core of experienced personnel can provide leadership and guidance to a staff of 
developers working on a version upgrade or patch. 


Ill 



THIS PAGE INTENTIONALLY LEFT BLANK 


112 



V. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 

Following is a brief discussion of each of the conclusions that arise from the 
research and analysis. The analysis of a specific program is required to determine issues 
and the associated required measures. For example, the research into possible metrics 
does not reveal measures associated with communicating program funding needs. The 
examination of system issues for CAMIN demonstrates the clear need for metrics in this 
area. 

The conclusions that follow should assist in providing statistical data to support 
funding justification. The organizations that justify and analyze funding for CAMIN 
require supporting documentation for these actions, but planning what data are required 
or requested in the future is rarely possible. A PM benefits from developing methods of 
easily capturing measurements for different aspects of the program. CAMIN uses 
databases to track action items, CCB PCR information, requirements baseline and 
changes, and the like. However, additional metrics are necessary if the program is to be 
supportable in the long-term. 

The CAMIN program requires software metrics and fielded system metrics. 

The analysis shows that the CAMDDN PM must use metrics from both software and 
from fielded systems to address primary programmatic issues. Key areas not addressed 
in software metrics are customer satisfaction and availability. Both of these areas are 
essential to evaluation and justification of fielded systems. 
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The CAMIN program requires metrics that are specific to the program, 
utilizing standard and non-standard metrics. 

The CAMIN system needs metrics that are not standard to software or fielded 
systems. One example is a metric to address communication of funding requirements. 
The funding sources for CAMIN are inexperienced in software systems, and are 
unfamiliar with the iterative nature of PDSS management. 

The PMO must continually evaluate metrics for the life of the system. 

The program issues change through the life of a system. Political, policy and 
requirements changes are continuous, and drive program issues. Issue changes require a 
review of program metrics. 

A model for establishing metrics that this thesis uses is depicted in Figure 9. The 
first step is to establish issues, through a thorough system analysis, including evaluation 
of the program’s internal and external environment. The issues should also consider 
program strengths, weaknesses, opportunities, and threats. After issues become clear, the 
issues must be refined into fundamental areas of measurement. The metrics and 
measures of each issue must be established, along with methods for data collection and 
analysis. Implementation and data collection provide the data needed for evaluation. 
The evaluation process may yield changes to system management, operations, or 
measures. 

These changes must be fed back into the metrics process to maintain a current 
system of measures. Finally, the PM must reevaluate the entire process of establishing 
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the program issues that drive metrics. The reevaluation should occur annually, 
concurrent with revision to the program strategic plan. 



[Developed by Researcher] 

Figure 9. Model for Selecting Program Metrics 


Management Information Systems are unique. 

Management Information Systems requirements change frequently, affecting 
overall program management and funding. These systems manage financial, office. 
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property, and other office-oriented missions. Drivers for the requirements changes 
include external environment, policy, user group, and COTS. 

Each Helded software system may require unique metrics. 

The metrics required for CAMIN, at this point in the life cycle, may be different 
from other fielded software systems. Each fielded software system PM must study 
program issues to develop a set of metrics for that particular program. Program managers 
should not rely on published standards for metrics. 

B. RECOMMENDATIONS TO CAMIN PROGRAM MANAGER 

The CAMIN PM should adopt the set of metrics in Table 11 for man a gin g the 
CAMIN program. These metrics are the correct metrics for the system, as it exists today. 
The measures derived through application of these metrics should reside in a central 
repository where all who are involved in the program management process of CAMIN 
can access and use the data collected. 

The PMO should reassess these metrics and data each year, using the process 
described in Figure 9. Reassessment should occur in consonance with the publishing of 
the Annual Business Plan for CAMIN. The reassessment should look at the cost of 
collecting the data, as a factor in assessing the value of the measure. The PM should 
revise the Annual Business Plan to reflect the current metrics and results to date. The PM 
should use the process depicted in Figure 9 in the analysis. 
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The lessons learned from this research, described in Chapter IV of this thesis and 
listed here, become additional recommendations to the PMO. The PM should strive to 
achieve these actions, within the control and purview of the organization. 

• Ensure that decision makers fully imderstand the CAMIN program 
requirements (software, hardware, people, dollars). 

• Ensure that users and their supervisors understand and take responsibility for 
having competent (trained and prepared) CAMIN system users. 

• Ensure that mission representatives understand the importance of data audits. 

• Maintain knowledge of trends and pending actions in the external IT 
environment, for both Government and industry. 

• Work toward Thin Client Architecture for CAMIN. 

• Achieve a constant workload for the CAMIN contractor to retain experienced 
IT support. 

C. RECOMMENDATIONS TO OTHER PDSS MANAGERS 

PDSS managers who are interested in defining appropriate metrics for their 
program should follow the same template as was used in this thesis, as shown in Figure 9. 
Developing a list of issue areas for the unique fielded software system is the first step in 
this process. For systems where issues are common to CAMIN, the analysis is complete. 
For different issues, a review of the candidate metrics may help to develop a tailored list 
of metrics for a fielded software system. During the analysis, PDSS managers should 
keep an open mind to new metrics that may fit their particular program and its issues. 
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D. FUTURE RESEARCH 

This section looks at areas that warrant future research. The areas of PDSS and 
metrics bring a wide range of ideas for research, encompassing the metrics for CAMIN 
program and other programs in the PDSS phase. Additionally, there is a wide range of 
other research to be done in the study of managing fielded software systems. This area is 
becoming more important as more software systems are fielded. 

1. Evaluate Outcome from Metrics Selected on CAMIN Program 

An area of research that grows directly from this thesis is assessment of 
implementing the thesis recommendations. The analysis would contrast the value of the 
metrics against the imagined benefit, assess the measures against the five issue areas for 
the reporting period, and evaluate the cost of the implementing and using these metrics. 

2. Evaluate Selected Metrics for Other PDSS Programs 

Another area that relates to this thesis is to assess how another fielded software 
system could use the outcome of this thesis. This action would include measuring the 
value of the metrics against the imagined benefit. It may also compare the results 
identified for the other fielded software systems, and compare to the results on the 
CAMIN program. 

3. Evaluate Other Aspects of Fielded Software Systems 

Metrics is only one area of interest in the program management of fielded 
software systems. The issues uncovered during the course of case interviews and other 
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data collection require further analysis. This is a short list of ideas for future research 
into PDSS management. 

Research into the nature and necessity of requirements changes after fielding 
would be beneficial. The fi-equent changes to requirements for fielded software drive 
cost and workload for the life of the system. Many of the requirements changes for 
CAMEST are not foreseeable, so the results of such analysis would be instructive for PMs. 

The maintenance costs for PDSS seem imcommonly high to most Army 
managers. The annual costs run about 20 percent of the total development cost of a 
system. A case analysis of program management costs for PDSS Management 
Information Systems across Army or DoD would provide useful baseline data and 
analysis. 

Another area of research evaluates trends in system architecture and management 
that can improve system access, ease of use, and avoid cost through the system life cycle. 
As with hardware systems, the initial design for software systems can drive up costs in 
maintenance and in logistics support. The DoD approach is to maximize use of COTS 
and standard programming languages. This research may support the value of the 
recommendations. 

Case research into PDSS to compare management techniques may provide insight 
into successful and unsuccessful approaches within DoD. The size, complexity, and 
visibility of the system play into the management techniques. For example, a system 
rated with a high acquisition category requires a classic PM structure within DoD, and 
has rigid requirements during the system development. After fielding, DoD regulations 
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are not as directive of system management requirements as those established for 
developmental systems. 

Finally, the study of retention of IT professionals on a fielded software systems 
can provide data and analysis that would address both DoD employees and contractors. 
The results may contribute to hiring practices and pay scale policies within DoD, and to 
IT contracting practices. The practice of acquiring and retaining qualified, capable, and 
consistent IT professionals is the core of a successful development and sustainment 
program. 
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APPENDIX A. CORRESPONDENCE 


The author requested these data from the managers of systems considered for the 
case interviews. The author conducted Case Interviews on phone and in person. 

Full Name of System 
Type of System 
Mission 
Scope 

Size (SLOCs) 

Architecture (very general) 

Number and types of users and connectivity 

Brief History of development and versions 

Metrics and Measures, and other program management Tools 
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APPENDIX B. CAMIN WORKSTATIONS AND USERS 
[After Ref. 14] 


LOCATION 

FACILITY 

FACILITY 

TYPE 

WKSTNS 

USERS 

WEB 

ONLY 

USERS 

Aberdeen, MD 

Aberdeen Chemical 
Demilitarization Facility 
(ABCDF) (Under Construction) 

CW Destruction 

0 

0 

Aberdeen, MD 

Edgewood Chemical Activity 
(ECA) (Active Stockpile Site) 

CW Storage 

3 

2 

Aberdeen, MD 

Edgewood Chemical Biological 
Center (Active Operations) 

Schedule 1 

Single Small- 
Scale Facility 
(SSSF) 

2 

3 

Aberdeen, MD 

Program Manager for Chemical 
Demilitarization (PMCD) 

Management and 
Operations 

2 

5 

Aberdeen, MD 

Soldier and Biological Chemical 
Command (SBCCOM) 

Management and 
Operations 

4 

9 

Anniston, AL 

Anniston Chemical Activity 
(ANCA) (Active Stockpile Site) 

CW Storage 

5 

2 

Anniston, AL 

Anniston Chemical 
Demilitarization Facility 
(ANCDF) (Under Construction) 

CW Destruction 

0 

2 

Commerce City, 
CO 

Rocky Mountain Arsenal 
(Active Destruction) 

Former CWPF 

2 

0 

Dugway, UT 

Dugway Proving Ground 
(Active) 

CW Storage 

2 

2 

Fort Leonard 
Wood, MO 

Fort Leonard Wood 10 kg 

Facility (Active Operations) 

Schedule 1 10 kg 
Facility 

1 

2 

Hawthorne, NV 

Destruction Facility 
(Intermittent, —Currently 

Inactive) 

CW Destruction 

0 

0 

Johnston Island 

Johnston Atoll Chemical Agent 
Destruction System (JACADS) 
(Active, almost complete) 

CW Destruction 

4 

6 

Lexington, KY 

Blue Grass Chemical Activity 
(BGCA) (Active Stockpile Site) 

CW Storage 

4 

0 

Lexington, KY 

Blue Grass Chemical 
Demilitarization Facility 
(BGCDF) (Planned) 

CW Destruction 

0 

0 


123 

































































LOCATION 

FACILITY 

FACILITY 

TYPE 

Newport, IN 

Newport Chemical 

Demilitarization Facility (NECDF) 
(Under Construction) 

CW Destruction 

Newport, IN 

Newport Chemical Depot (NECD) 
(Active Stockpile Site) 

CW Storage 

Ne\vport, IN 

Newport Chemical Depot (NECD) 
(Active Destruction) 

Former CWPF 

Pine Bluff, AR 

Pine Bluff Chemical Activity 
(PBCA) (Active Stockpile Site) 

CW Storage 

Pine Bluff, AR 

Pine Bluff Chemical Activity 
(PBCA) (Active Destruction) 

Former CWPF 

Pine Bluff, AR 

Pine Bluff Chemical 
Demilitarization Facility (PBCDF) 
(Under Construction) 

CW Destruction 


WKSTNS 

USERS 


Pueblo, CO 


Pueblo, CO 


Tooele, UT 


Tooele, UT 


Tooele, UT 


Umatilla, OR 


Umatilla, OR 


Washington, DC 


Facility (PUCDF) (Planned) 


Pueblo Chemical Depot (PCD) 
(Active Stockpile Site) 


Chemical Agent Munitions 
Destruction System (CAMDS) 
(Active)_ 


Deseret Chemical Depot (DCD) 
(Active Stockpile Site) 


Tooele Chemical Demilitarization 
Facility (TOCDF) (Active) 


Umatilla Chemical 
Demilitarization Facility 
(UMCDF) (Under Construction) 


Umatilla Chemical Depot (UMCD) 
(Active Stockpile Site) 


Department of Army 


CW Destruction 


CW Storage 


CW Destruction 


CW Storage 


CW Destruction 


CW Destruction 


CW Storage 


Management Only 



Washington, DC 

US Army Nuclear and Chemical 
Agency 

Management Only 

0 

1 

Washington, DC 

Defense Threat Reduction Agency 

Management Only 

0 

3 

Honolulu, HI 

US Army Pacific (USARPAC) 

Management Only 

0 

1 


TOTALS 


53 


68 


























































NT Applications 


APPENDIX C. CAMIN LINES OF CODE 


[From Ref. 4] 


Version 8 Components 


Ad Hoc 


Alert Catalog 


Version 8 Version 7.5 Version 7 


CWPF Scheduling 
Data Administration 
Declaration Catalog 
Declared Chemical 
Declarations 
Data Download 
Material Release 
Notification Catalog 
Notifications 
Notify Server 
RAS Manager 
Reports 
Schedule 1 
Schedule 2 , 3, Other 

Site Diagrams _ 

Site Information 
Stock Records 
User Administration 
NT Total 



Annual Inventory 



CAMIN DLL 


87361 

CAMIN Settings 



CAMIN Setup 

2548 

2641 

Compare 

6063 

6162 

CW Destruction Reporting 

0 

9352 

CW Destruction Scheduling 

0 

11731 


CW Items 


CWPF History 

2i 



23348 

565 


3265 


124 

27 


9274 


2477 


3086 


2487 


29667 


22122 


7952 


16458 


59516 


372603 


24326 


0 


3250 




1 4445 

4687 

1997 

1934 


27873 

22345 

22321 

0 

0 

21111 

21073 

36435 

37437 

9903 

10318 

350869 

346416 

































































Server Applications 


Version 8 Components Version 8 Version 7.5 Version 7 


Chlst 

58: 

52{ 

473 

chlst_lib 

1988i 

jEHQE 

9110 

combine 




compresslib 

467f 

4695 


dbupdate 

IHIKS 

IHHKSi 

390 

Dcelib 


lEBKEiE 

4492 

decl_lib 

1041S 

9813 

7787 

decl_ipts/decl_cw_dest 

23462 

23458 


declrpts/declcwinitial 

19091 

19091 


decl_rpts/decl_cw_schedl 

0 

7752 

13635 

decl_rpts/decl_cw_sched2_3_doc 

52036 

52034 

21657 

decl_rpts/decl_cwpf_ann_dest_rpt 

5224 

5220 

3333 

decl_ipts/decl_cwpf_ann_dest_sched 

HHH 

16483 


decl_ipts/decl_cwpf_initial 


17002 


decl_rpts/decl_view_comments 

1162 

1162 


delete_elements 


3212 

1723 

dispatcher 


3615 

3111 

extemal_trans 

0 

0 

2605 

file_server 

eib^ 

1911 

1825 

fwd_n_rej ect_docs 



5512 

ing_lib 

2090 

2090 

2714 

intemal_trans 

0 

0 

3156 

issue-docs 

1470 

1405 


Lib 



1922 

localdata 

137 


135 

planagraph 



3418 

post_docs 


lE^^I 


q_combine_msg 




qa_process 

10447 

4997 

3254 

Server_lib 

1822 

^mm\ 

1448 

sqlbatch 

37613 


12870 

stock_trans 



0 

slashgo 

76 

76 

76 


141 

141 

142 


272 

272 

267 

add_server_users 

324 

324 

354 


126 









































Version 8 Components_Version 8 Version 7.5 Version 7 



caminarchive 

51 

57 

57 


camin-files-clean 

4063 

4555 

0 


camin_shutdown 

57 

57 

57 


camin_test 

294 

294 

0 


check_backup 

374 

374 

li^MI 


check_backup_fs 

386 

772 

i^Sl 

e 

clr_rtn 


107 

|||||^■|■^ 


dcewho 

262 

262 

262 

68 

W 

del_dce_users 

105 

94 

95 

a 

a 

dirlist 

0 

0 

0 

< 

u 


52 

52 

0 

© 

t 

list_dce_users 

233 

233 

233 

© 

05 

mailbox_test 

169 

174 

0 


procinfo 

225 

225 

0 


sho_db_users 


140 

141 


sho_queues 

46 

46 

46 


include 

8202 

7669 

10358 


HA probes 

3100 




Server Total 

344462 

319762 

212917 


Version 8 Components 

Version 8 

Version 7.5 

Version 7 


CAMINHelp 

9641 



C 

© 

HTML 

5662 

632 


pO 

68 

Servlets 

17339 

13243 

0 

■ 

Applets 


43435 

0 

a 

< 

Web Total 

77173 

57310 

0 
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Version8Components _Versions Version7.5 Version? 



CW Inventory Reports 



4300 


CWPF Reports 


1900 

1900 

pS 

Weapons & Agent Reports 


IHHES 

900 

& 

s 

Installation Reports 

700 


700 

u 

H 

Permission Reports 


600 

600 

u 

o 

Request Reports 


200 

200 

a 

Schedule 1 Reports 

1100 

1100 

1100 

K 

Oracle Reports 


2100 

0 


Reports Total 


12000 

9700 


Version 8 Components 

Version 8 

Version 7.5 

Version 7 


Tables 

3410 

2674 

2561 


Views 

3294 

2010 

2010 

es 9 

indexes 

620 

510 

823 

« 2 
ts ? 

grants 

422 

335 

335 

w is 

e X 

synonyms 

422 

328 



Backup scripts 

2113 




Database Total 

mmm\ 

5857 

5729 


- ——— --- ■ 

E 

System Totals 

831519 

745798 

574762 
























APPENDIX D. CAMIN CONTRACTOR METRICS 


[From Ref. 3] 


Requirements 

❖ Requirements Identified 

❖ Requirements fulfilled 

❖ Requirements outstanding 

❖ Requirements validated (for a release) 

❖ Requirements by release 

*t* Requirements by Configuration Item 

❖ Requirements by category (new, baseline change, design 
change, implementation change) 

❖ Changes per configuration item (volatility) 

Program 

Management 

❖ Cost to complete 

❖ Hours expended by day, month, task order/project 

❖ Hours to complete 

♦♦♦ Cost expended by month, task order/project 

❖ Baseline (cost/hours/duration) vs. actual & progress (% 
complete) 

❖ Software size by version (SLOC) 

❖ Software size by tasking (function point) 

Quality Assurance 

❖ Quality Audits planned/conducted 

❖ Nonconforming conditions found 
♦♦♦ Nonconforming conditions closed 

❖ Open nonconforming conditions. 

Configuration 

Management 

❖ Backup success 

❖ Backup resources used 

❖ Hardware Environment Changes 

❖ Software Environment Changes 

❖ Number of files checked out 
^ Number of files changed 

♦♦♦ Number of users who changed a file 

❖ Number of files changed 

❖ Number of times each file was checked out 

❖ Number of users who checked out a file 

❖ Number of users who changed a file 

❖ Number of lines that changed per file (code only) 

❖ Average number of changed lines (code only) 

♦♦♦ Average length of time a file was checked out 

Training 

❖ Training Requirements, vs. training fulfilled 

❖ Effort/hours expended in staff familiarization (training) 
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Peer Review 

Peer Reviews planned/conducted 

❖ Problems/Defects identified 
♦♦♦ Problems/Defects closed 

❖ Time to Closure 

Critical Resoiirce 

❖ Staffing Needed by Lifecycle 

❖ Equipment resources needed by Lifecycle 

Maintenance 

Problem Calls logged (by category, by users, by action) 
Time/effort to service calls 

❖ Open problem calls 

❖ Data changes/interventions logged 

Time/effort to make data change 

Development 

lime to implement (baseline vs. actual) by software 
change/Configuration Item 
♦J* Defects/Problems identified 
♦♦♦ Defects/Problems corrected 
♦♦♦ Defects/Problems validated 
❖ Time to closure (date corrected -date identified) 

♦♦♦ Effort to correct 

Intergroup 

Coordination 

❖ Action Items logged 

❖ Action Items open 

❖ Time to close items 
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APPENDIX E. CAMIN SCREEN CAPTURES 


This appendix shows samples of screen captures from the CAMIN system. The 
data displayed in these screens is from the “training” database, and are likely to be 
invented data. This rudimentary hierarchy of the CAMIN interface helps explain how the 
following screens fit into the larger CAMIN system. 


CAMIN Applications 

CAMIN Web Site 

• General Data 

• Calendar 

o Declared Chemical Information 

• Report Generator 

o Chemical Weapons Items 

• Notification Archive 

o Site Information 

• Site Diagrams 

■ Installations and Plant Sites (Site 

• Process Flow Diagrams 

Diagrams) 

• Newsletters 

■ Declared Facilities and Plants 

• rrR Info 

■ Building Information (Planographs) 

• DMUG Info 

• Data Access 
o Alerts 

• Software Updates 

o Data Download 

• Training Information 

o Site Diagram Modifications 

• Policy Information 

o Reports Viewer 

• Documentation 

• Stock Records 

• Help 

o Stock Records Maintenance 

• Administration 

o Materiel Release Information 

• Links 

o Document Register 


o Annual Inventory 


o CW Destruction Reporting 


• CWPF History 


• Schedule 1 Permitted Activities 


• Administrative Tools/User Administration 


• Common Tools 


o CAMIN Settings 


o Data Administration 


o Ad Hoc Query 


• Notifications and Declarations 


o Notifications Generation and Coordination 


G Notifications Catalog 


o Declarations Generation and Coordination 


o Declarations Catalog 
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Site Information > Installations and Plant Sites 




fie £(& j2ata jgenerellrrfo fieporte QA )£/hckm Hefc 



Name: |Blue Grass CW Storage Facffity 


Type: 

Status: 

Latitude: 

ton^lude: 

TSCode: 

Pctcent Capdd^ {Zj; 


g Facili^ 


jActive 

j37/43/08N 

j084/13/00fV/ 

JlSCFac-B 

I mSS 


~3-- 


: Commandef Blue Grass Chemical Activily 2091 
= Kingston Highway 


zi 


C%: )Richmond 


Slate: jKentucky 




Zip: [40475-5008 ^ 

Cowty (urtted S tates ot America | 


li^al^m [Blue Grass Chemical Activity 




Owner |mAJ JohnM. Riley 
Operator: [mAJ JohnM. Riley 


Type of Last Inspection: jRoutlne CW Storage 

,1^^ Insp infa, | 


Eoewo... 


Name 


Deborah L Boston 


I 


Office 


I Function : J. . Office Phone I-' 


AMSSB-OBG-TO Treaty Officer 859-825-6285 


lya03 itrain lUPOATE |OPEN INONE INONE 


.NUM ; 


The screen shown here displays the types of data CAMIN stores for an 
installation. CAMIN also has a screen for each facility that resides within an installation 
or plant site. CAMIN workstation users, with permission, can change the information. 
These data are used by both treaty and accoimtability applications in CAMIN. 
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Site Information > Building Information (for Storage facility) 


Site Information < [Building Information] 


Fie £cR Data Gen^^lnfo Reports §A 'Vtm Hdp 


^X] 



;U.P0AT|.; lOPENiNONE . JNONE i -NUM j A 


This screen shows an example of a building list in CAMIN for a storage facility. 
Each building may have unique grid layout that users would establish within this 
application. The first building on the list is currently imdergoing annual inventory. The 
Annual Inventory application in CAMIN allows scheduling, printing out inventory 
worksheets, entering the completed inventory, and notifying the accountable officer of 
completion. 




































































Site Information > Building Information (for CWPF facility) 


-^£fe £<it Data £enerat Info Reports flA View Window 


^:^Site Information - [Building Information] 


HIMD 





guiding Info..; 
POCittfft.. } 
Btldlrrfo.;. I 


i 8u3cSna/Slructure 

1 ID" 

1 Status 

i Tui:»* 

i pnr 

1 


Analyzer Shed 

! Analyzer Shed 

Inactive 

! Standard 

; None 

No 

DF Various 

; DF Various 

Inactive 

: None 

None 

No 

bF/M20 Building 

: 53-220 

Inactive 

Specialized 

None 

Yes 

MLRS Fill Building 

53-210 

Inactive 

Specialized 

None 

Yes 


|W.03 ftfain 


lUPDATE . lOreN ;N0NE InONE 


This screen shows a listing of buildings that would be associated with a former 
CW Production Facility (CWPF). The same application is used as for the last screen, but 
for a different purpose. Through reuse, CAMIN has saved development costs and 
avoided duplicate maintenance costs. 
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Planograph 



The Planograph is a map of a storage structure, showing grids where stock is 
stored. Each grid shown on the screen is listed on the next page, with the contents of that 
grid. Users enter the data used in creating the Planograph through the Stock Records 
Maintenance application. 
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This screen represents the second page of a Planograph. This listing shows the 
contents of the structure by grid location. The Planograph is beneficial in annual 


inventories, maintenance activities, emergency response, and in treaty inspections. 
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Chemical Weapons Item Information 




mmm 


^ £fe 

Info;: Repoi 

ts SA View Window _ljelp „ ' 



14 1 4 1 1 M 1 9 1 1 




Commwi Name of ;Ch«i^ W IUnknown 


Type: . : p 

Oesgn^on ‘ , Ji 

Size/CaSber p 

FiWeight f 

F® Volume [cam^ f* 

OeslfucticBi Categc^ p 

treafyCafegoiy: p 

Nomend^uer p 

Declared under and BDA: 
BiltCdr^aBTen n 


jBomb 

jM47[SusH) ^ 

pOOlb ^ 

I 27.22000 . 

p 1023000000 

[category 1 (Sc^e^ 1 Fill] 
[Munition 

|UNK Bomb M47(SusH) 1001b 


MARB Numbers 


MARS mfa,.' 



Rea(» . . ; |TOb3 Hrem iMPDATE ;0PEM (METRIC jKilogiams { ^ 


This application provides a dictionary of all CW items in the CAMIN system. 
The application creates a structured format for all nomenclatures, in accordance with the 
Chemical Weapons Convention. The application also creates a cross reference between 
nomenclatures and National Stock Numbers, to conform to the Army cataloging system. 
The nomenclature consists of four elements: 1) the name of the chemical fill, 2) the type 
of item, 3) the military designator, and 4) the size or caliber of the item. “None“ or 
“unknown” are acceptable for almost every field. 
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Declared Chemical Information 



ge Edit Data General Info fiepoHs C|A yiew jj/h dow Heis 

MiHlal * ' M l 1 : 



Record 


GannKMi Name: ■ piBHgMiSBM' 

lUPAC Water and Bleach 

Npnr^ndature: 


Designator W^^ 


Schecye: 


I Unscheduled 




CAS Registry #: iNone-WB 


IS Code: 


Sp^c^c Gf^ily: 


|TSCChem-50 


1.23457 


Toxid^ 

Prim^^ 

Re^tion 

Product 


J very toxic 



j^ain JUPDATE pDPEN jNONE [TJ 


Each chemical fill in CAMIN is defined in this application. The chemical must 
be defined before a CW Item can contain that fill. 

Both the CW Items and the Declared Chemicals Applications perform as 
dictionaries for CAMIN. Any change to these screens may affect multiple items located 
at multiple sites. 
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Stock Record Card for Chemical Items 




ffe £dft General Info Beports fiA He^P 


BlHla^l ll^liii H H !► I H I fi 1 




foslalaiion: ■ [besefet ChemicalJDeiM^ 

Dedared Fsc^, |Des«et CW Staage Fadtty 




3 


Record 


1 of 


2-' M 


Nomenctehie: jGB OneJa^Ca^rT^c D ^ ^ 7. 

SlockNunnber |l3^2939239 ^ ^ ^' 


Lol Number jRM76033-:K7 ^ 


13 


i 


Treaty 
location ^ 


Transachon 

Date 


Transadion 

Status 


Custocial 

IdCatioh 




cr 


Stock 


Amount 

Agent 


] i:i?-Apr-2G01 ATR Pending-OAR eq 1103 TOCDF D-84160 AFAE 


I A i: 


Die 


Docun«t^ 
: Number 


5081 


oeso;: AIR M TOCDF-01097-1004 


■LmJ— 


Cj 


StodcBaiance: | 
AgentBalance: | 


Rein»ks; 


IA1103P9726812 


.880388987 


3 


Read) 


jva03 jlrain :UPOATE iDPEN iMETRIC iTotmes ’ jNUM f A 


This screen displays a single lot of an item, stored at a single facility. From this 
screen, the user with appropriate permissions at the facility can change characteristics of 
items or move items to another grid, building, facility, or installation. 
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Document Associated with Transaction 



The CAMIN creates documents that are required by Army regulation for 
management of wholesale Army property. Regulations require these documents for 
changes to status or shipments. 
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Data Download 



This application was not an original system requirement. It was part of a project 
to compensate for slow network connections at the Johnston Island facility. The Data 
Download application populates the lookup tables that reside on CAMIN workstations. 
Lookup tables contain data from the central server that changes infrequently. The 
CAMIN software uses the lookup tables to populate screens without receiving all data 
from the central server. 
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■BSSSH 




Current Version: 8.03 


HOME , CALENDAR ; DATA ACCESS f SYSTEM INFO SOFTWARE UPDATES i TRAINING ; HELP i ADMIN 


CAMlNHeip Line: 4Z0~436-3lS9 
CAMm Pager: 1-$00~SKY~8888 ('WV 1714303) 

Send cfTtaii to tfte P-sger ^ i I 


• Home 

• Calendar 
•Report Generator 

• Notification Archive 
•Site Diagrams 
•Process Flow Diagrams 
•Newsletters 

•CCB Info 
•DMUG Info 

• Software Updates 

• Training Information 

• Policy Information 

• Documentation 
•Help 

• Administration 

Links 

SBCCOy/ Home Page 
U.S. Army Home Page 
OPCIV Home Page 
PMCD Home Page 
PNCD CWC Net 


Announcements 

No current ann€>uncements 


: Ani^tlTcemerTtAn^^ j 



Meetings 

Training 

No oirrent ^innounc0n^Bnts 

• CAMIN Classroom Training for 

CWPF and Schedule l 17 - 19 April. 

'^G^ehdar 1 HMeebmi Archive 1 

ilMorelrifo i 



To provide feedback to the webmaster, click here . 


- i; it® Irtemet 




The CAMIN Web Site provides access to commonly needed data from the 


CAMIN server. The data presented by this web site is from the same source as the client 
system. The web site users can also get programmatic information, training, help, and 
calendar information. 


Ultimately, this web site will support an increasing portion of the CAMIN client 
(workstation) capability. In this way, the PM can reduce the dependence on workstations 
and increase system access. 
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